Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Addition: Add support for Hypra and EthashB3 #617

Draft
wants to merge 44 commits into
base: master
Choose a base branch
from

Conversation

JABirchall
Copy link

@JABirchall JABirchall commented Feb 7, 2024

This pull request is draft for now as we go through the steps to add support for Hypra network.

Hypra is a custom Proof-of-Work chain based on Ethereum, but recently we have had difficulty to upgrade the chain with upstream ethereum features. We want to include these changes in core-geth to make ethashb3 available for use in other chains and allow for easier upgrade of the Hypra network.

https://www.hypra.network/blog/the-future-of-geth-and-hypra.html

Our end goal is to add support for Hypra network to core-geth under the --hypra flag, and maintain our own version where Hypra is the default network, pulling down upstream updates and pushing updates when hypra adopts a fork.

Our current geth code avalible here: https://github.com/Rethereum-blockchain/go-rethereum

Merge checklist

  • Support EthashB3
  • Support Hypra Genisis
  • Support Hypra existing forks
  • Support custom mining reward
  • STATUS: EthashB3 dev chain working
  • STATUS: Syncing Hypra Mainnet
  • STATUS: Producing blocks on Hypra Mainnet
  • STATUS: Ready to merge

@JABirchall
Copy link
Author

Currently Hypra supports EVM upto Shanghai.
Hypra does not support EIP-1559 -- This is planned to fork off of the main geth client and switch to core-geth. Or cancun.

@JABirchall
Copy link
Author

JABirchall commented Feb 8, 2024

Update, I found where to define ethashb3 engine creation.

IMO, i felt the CreateConsensusEngine was very busy and took it on my self to refactor that part so each engine is created in its own function and doesnt need more parameters passed through to 1 function, it now switches from the chainspec. This makes it easier and cleaner to add new censensus engines in future.

Feel free to review it, i can revert the change if you decide its not appropriate. Ive kept the orignal CreateConsensusEngine for legacy reasons and some part of the codebase still use it.

@meowsbits

6bea66a

@JABirchall
Copy link
Author

Just confirmed ethashb3 is now working and useable in a chain configuration.

@JABirchall
Copy link
Author

JABirchall commented Feb 11, 2024

Ok, im 99% sure ive completed everything needed for Hypra to be compatible. However, upon testing. CoreGeth cant sync from a Geth node which all our current nodes run on.

If i copy the chaindata into the coregeth directory it starts with all the blocks and another coregeth node can sync from that. But they are not able to connect to any geth nodes to stay in sync.

Any ideas @meowsbits this would be needed for transition from geth to core-geth.

@JABirchall
Copy link
Author

JABirchall commented Feb 12, 2024

From some testing ive done

  • Hypra initializing ethashb3
  • Hypra genesis is the correct hash
  • Hypra does not sync from mainnet geth nodes
  • EthashB3 throws "unable to delete cache, file in use" errors

@meowsbits
Copy link
Member

@JABirchall

CoreGeth cant sync from a Geth node which all our current nodes run on.

Might be a Fork ID problem? On quick glance I didn't see any fork id files changed here yet...
If you use --log.verbosity 5 you might be able to see the error to get at what's causing the failed conn.

@meowsbits
Copy link
Member

If i copy the chaindata into the coregeth directory it starts with all the blocks

Just FYI, you could also use geth import to copy the (geth exported) chain data, which will have an added benefit of being a full regression test.

@JABirchall
Copy link
Author

JABirchall commented Feb 13, 2024

core-geth Hypra is now fully syncing with the network!

@JABirchall
Copy link
Author

JABirchall commented Feb 13, 2024

How can I build core-geth for linux and mac?
My normal build script cant build coregeth saying Lyra2 is undefined.

Builds fine on Windows

GOROOT=C:\Program Files\Go #gosetup
GOPATH=C:\Users\owner\go #gosetup
"C:\Program Files\Go\bin\go.exe" build -trimpath "-ldflags=-s -w" -o C:\Users\owner\Documents\projects\core-geth\bin\go_build_release_linux . #gosetup
github.com/ethereum/evmc/v7/bindings/go/evmc: build constraints exclude all Go files in C:\Users\owner\go\pkg\mod\github.com\ethereum\evmc\v7@v7.5.0\bindings\go\evmc
# github.com/ethereum/go-ethereum/consensus/lyra2
..\..\consensus\lyra2\api.go:15:9: undefined: Lyra2
..\..\consensus\lyra2\sealer.go:207:16: undefined: Lyra2
..\..\consensus\lyra2\consensus.go:50:14: undefined: Lyra2
..\..\consensus\lyra2\consensus.go:56:14: undefined: Lyra2
..\..\consensus\lyra2\consensus.go:77:14: undefined: Lyra2
..\..\consensus\lyra2\consensus.go:139:14: undefined: Lyra2
..\..\consensus\lyra2\consensus.go:157:14: undefined: Lyra2
..\..\consensus\lyra2\consensus.go:213:14: undefined: Lyra2
..\..\consensus\lyra2\consensus.go:276:14: undefined: Lyra2
..\..\consensus\lyra2\consensus.go:334:14: undefined: Lyra2
..\..\consensus\lyra2\consensus.go:334:14: too many errors

@JABirchall
Copy link
Author

Had a good run for 1 hour. But seems somewhere a a gas cost is different.

ERROR[02-13|23:45:13.652] 
########## BAD BLOCK #########
Block: 2059835 (0x5d7372b0590e19ae733c72f2f17a1ef53a9c716b328c190a3fdcfb88e36fa652)
Error: invalid gas used (remote: 178816 local: 208973)
Platform: geth (devel) go1.21.1 amd64 windows
VCS: 5a26d5db-20240213 (dirty)
Chain config: &coregeth.CoreGethChainConfig{NetworkID:0x97ec5, ChainID:622277, SupportedProtocolVersions:[]uint{0x44, 0x43, 0x42}, EIP2FBlock:0, EIP7FBlock:0, DAOForkBlock:<nil>, EIP150Block:0, EIP155Block:0, EIP160FBlock:0, EIP161FBlock:0, EIP170FBlock:0, EIP100FBlock:1001,
 EIP140FBlock:1001, EIP198FBlock:1001, EIP211FBlock:1001, EIP212FBlock:1001, EIP213FBlock:1001, EIP214FBlock:1001, eip649FInferred:false, EIP649FBlock:<nil>, EIP658FBlock:1001, EIP145FBlock:5503, EIP1014FBlock:5503, EIP1052FBlock:5503, eip1234FInferred:false, EIP1234FBlock:<
nil>, EIP1283FBlock:5503, PetersburgBlock:5507, EIP152FBlock:5519, EIP1108FBlock:5519, EIP1344FBlock:5519, EIP1884FBlock:5519, EIP2028FBlock:5519, EIP2200FBlock:5519, EIP2200DisableFBlock:<nil>, eip2384Inferred:true, EIP2384FBlock:5521, eip3554Inferred:false, EIP3554FBlock:1
3524557, eip4345Inferred:false, EIP4345FBlock:<nil>, EIP1706FBlock:<nil>, EIP2537FBlock:<nil>, ECIP1010PauseBlock:<nil>, ECIP1010Length:<nil>, ECIP1017FBlock:<nil>, ECIP1017EraRounds:<nil>, ECIP1080FBlock:<nil>, ECIP1099FBlock:<nil>, ECBP1100FBlock:<nil>, ECBP1100DeactivateF
Block:<nil>, EIP2315FBlock:<nil>, EIP2565FBlock:5527, EIP2718FBlock:5527, EIP2929FBlock:5527, EIP3198FBlock:1600957, EIP4399FBlock:<nil>, EIP2930FBlock:5527, EIP1559FBlock:<nil>, EIP3541FBlock:13524557, EIP3529FBlock:1600957, EIP5133FBlock:<nil>, eip5133Inferred:false, EIP36
51FTime:(*uint64)(nil), EIP3855FTime:(*uint64)(nil), EIP3860FTime:(*uint64)(nil), EIP4895FTime:(*uint64)(nil), EIP6049FTime:(*uint64)(nil), EIP3651FBlock:<nil>, EIP3855FBlock:1600957, EIP3860FBlock:1600957, EIP4895FBlock:<nil>, EIP6049FBlock:<nil>, EIP4844FTime:(*uint64)(nil
), EIP1153FTime:(*uint64)(nil), EIP5656FTime:(*uint64)(nil), EIP6780FTime:(*uint64)(nil), MergeNetsplitVBlock:<nil>, DisposalBlock:<nil>, Ethash:(*ctypes.EthashConfig)(nil), EthashB3:(*ctypes.EthashB3Config)(0x7ff6d752a680), Clique:(*ctypes.CliqueConfig)(nil), Lyra2:(*ctypes
.Lyra2Config)(nil), IsDevMode:false, TerminalTotalDifficulty:<nil>, TerminalTotalDifficultyPassed:false, TrustedCheckpoint:(*ctypes.TrustedCheckpoint)(0x7ff6d5304260), TrustedCheckpointOracle:(*ctypes.CheckpointOracleConfig)(nil), DifficultyBombDelaySchedule:ctypes.Uint64Big
MapEncodesHex{0x1591:9000000}, BlockRewardSchedule:ctypes.Uint64BigMapEncodesHex{0x0:4000000000000000000, 0xce5e4d:3000000000000000000, 0x19f0ab1:2000000000000000000, 0x26d6a73:1000000000000000000}, RequireBlockHashes:map[uint64]common.Hash{0x26160:0x2a27bec023108c5f650cb0c9
b7aaae7e7fdeefbeb3cd14a8390bb0957043aca2, 0x7d000:0xd547c896967174295c61ea18534d3c69337a6fa6e957909dc29e4bcba873592a, 0x1ca91b:0x1834091726bd0890539db04a22b673a6d9262d8b350dffaefdb10abc72e2113d}, Lyra2NonceTransitionBlock:<nil>, HIPVeldinFBlock:500009}
Receipts:
  0: cumulative: 208973 gas: 208973 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xb442721e1c3d4bd79fdacd396acec2581c44b8e3af71f085b9258e3073098790 logs: [0xc0014842c0 0xc001484370 0xc001484420 0xc0014844d0 0xc001484580 0xc0014846e0 0xc001484790 0xc0014
84840 0xc0014848f0] bloom: 00000000000000000000000081000000000000000000000000000000000000000000000000000000000100002000000000000000000000000000000000200000000000000000000000000008000000000400000000400000000000000000000000080004020000000000000000000800000000000100040000000010
000000000000000000000000000020000800080080000004000000080000000000000000020000000800004000000000000000000000000100000000000000000000000000000002000000400000000000000000000000000000001000000002000020000010000000000080000000000040000020000008000000000008000000000000 state:    
##############################

Cant figure out which EIP is responsable for the gas difference

@meowsbits
Copy link
Member

meowsbits commented Feb 15, 2024

This is confuing, since it made it past our last fork at 1,600,000~
So i am unsure whats causing the bad blocks. Especially since they are within the last 100 blocks needed to complete the sync

If using --syncmode=full or import it still fails on block 1

The --syncmode snap protocol will not validate all blocks; only a random fraction on the way up, and all of last few . That's why you're seeing inconsistent fails. I recommend testing block validation exclusively with --syncmode full and/or import.

How can I build core-geth for linux and mac?
My normal build script cant build coregeth saying Lyra2 is undefined.

Check the .github/workflows/ for build patterns. A go clean -cache and ensuring you're at the latest Go version never hurts.

Cant figure out which EIP is responsable for the gas difference

This is a tough problem. Many EIPs are gas cost adjustments. In order to debug you might try comparing block/tx trace results from the working client vs. the development one. With the A/B traces you can compare the block/tx opcode steps and their resulting states. Once you figure out which opcode step is mismatched, you can then cross-reference the target opcode against its associated gas-relevant EIPs.

@JABirchall
Copy link
Author

The --syncmode snap protocol will not validate all blocks; only a random fraction on the way up, and all of last few . That's why you're seeing inconsistent fails. I recommend testing block validation exclusively with --syncmode full and/or import.

Ultimately i figured out the problem by not syncing in mining block 1, and found that the block rewards were wrong because schedulereward would only return custom rewards for ethash, so i had to add ethashb3 to that and it worked.

This is a tough problem. Many EIPs are gas cost adjustments. In order to debug you might try comparing block/tx trace results from the working client vs. the development one. With the A/B traces you can compare the block/tx opcode steps and their resulting states. Once you figure out which opcode step is mismatched, you can then cross-reference the target opcode against its associated gas-relevant EIPs.

im doing a full sync right now to find what point in the chain gas difference happens and with core-geth it is rather slow. Its only syncing 100 blocks every 8 seconds. With standard geth it does 2048 blocks every 2 seconds.

@JABirchall
Copy link
Author

JABirchall commented Feb 16, 2024

Update: Sync is now 200k blocks off our gaspar fork without any issues so i think that narrows it down to something within the london/merge/shangai evm upgrades that were part of Gaspar

First block from Gaspar upgrade which i deployed a shangai test contract failed to sync

ERROR[02-16|13:47:25.794]
########## BAD BLOCK #########
Block: 1600994 (0xce5bbe1fb36a5b33e55591fa026b38d5e03a53c9e23686e8225b2b01e89ed645)
Error: invalid bloom (remote: 00000000000000000000000001000000000000000000000000800000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000020000000000000000000800000000800000000000000000000000400000000000040000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000  local: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000)
Platform: geth (devel) go1.21.1 amd64 windows
VCS: 5a26d5db-20240213 (dirty)
Chain config: &coregeth.CoreGethChainConfig{NetworkID:0x97ec5, ChainID:622277, SupportedProtocolVersions:[]uint{0x44, 0x43, 0x42}, EIP2FBlock:0, EIP7FBlock:0, DAOForkBlock:0, EIP150Block:0, EIP155Block:0, EIP160FBlock:0, EIP161FBlock:0, EIP170FBlock:0, EIP100FBlock:1001, EIP140FBlock:1001, EIP198FBlock:1001, EIP211FBlock:1001, EIP212FBlock:1001, EIP213FBlock:1001, EIP214FBlock:1001, eip649FInferred:false, EIP649FBlock:<nil>, EIP658FBlock:1001, EIP145FBlock:5503, EIP1014FBlock:5503, EIP1052FBlock:5503, eip1234FInferred:false, EIP1234FBlock:<nil>, EIP1283FBlock:5503, PetersburgBlock:5507, EIP152FBlock:5519, EIP1108FBlock:5519, EIP1344FBlock:5519, EIP1884FBlock:5519, EIP2028FBlock:5519, EIP2200FBlock:5519, EIP2200DisableFBlock:<nil>, eip2384Inferred:true, EIP2384FBlock:5521, eip3554Inferred:false, EIP3554FBlock:13524557, eip4345Inferred:false, EIP4345FBlock:<nil>, EIP1706FBlock:<nil>, EIP2537FBlock:<nil>, ECIP1010PauseBlock:<nil>, ECIP1010Length:<nil>, ECIP1017FBlock:<nil>, ECIP1017EraRounds:<nil>, ECIP1080FBlock:<nil>, ECIP1099FBlock:<nil>, ECBP1100FBlock:<nil>, ECBP1100DeactivateFBlock:<nil>, EIP2315FBlock:<nil>, EIP2565FBlock:5527, EIP2718FBlock:5527, EIP2929FBlock:5527, EIP3198FBlock:1600957, EIP4399FBlock:<nil>, EIP2930FBlock:5527, EIP1559FBlock:<nil>, EIP3541FBlock:13524557, EIP3529FBlock:<nil>, EIP5133FBlock:<nil>, eip5133Inferred:false, EIP3651FTime:(*uint64)(nil), EIP3855FTime:(*uint64)(nil), EIP3860FTime:(*uint64)(nil), EIP4895FTime:(*uint64)(nil), EIP6049FTime:(*uint64)(nil), EIP3651FBlock:<nil>, EIP3855FBlock:1600957, EIP3860FBlock:1600957, EIP4895FBlock:<nil>, EIP6049FBlock:<nil>, EIP4844FTime:(*uint64)(nil), EIP1153FTime:(*uint64)(nil), EIP5656FTime:(*uint64)(nil), EIP6780FTime:(*uint64)(nil), MergeNetsplitVBlock:<nil>, DisposalBlock:<nil>, Ethash:(*ctypes.EthashConfig)(nil), EthashB3:(*ctypes.EthashB3Config)(0x7ff75202b680), Clique:(*ctypes.CliqueConfig)(nil), Lyra2:(*ctypes.Lyra2Config)(nil), IsDevMode:false, TerminalTotalDifficulty:<nil>, TerminalTotalDifficultyPassed:false, TrustedCheckpoint:(*ctypes.TrustedCheckpoint)(0x7ff74fe14220), TrustedCheckpointOracle:(*ctypes.CheckpointOracleConfig)(nil), DifficultyBombDelaySchedule:ctypes.Uint64BigMapEncodesHex{0x1591:9000000}, BlockRewardSchedule:ctypes.Uint64BigMapEncodesHex{0x0:4000000000000000000, 0xce5e4d:3000000000000000000, 0x19f0ab1:2000000000000000000, 0x26d6a73:1000000000000000000}, RequireBlockHashes:map[uint64]common.Hash{0x26160:0x2a27bec023108c5f650cb0c9b7aaae7e7fdeefbeb3cd14a8390bb0957043aca2, 0x7d000:0xd547c896967174295c61ea18534d3c69337a6fa6e957909dc29e4bcba873592a, 0x1ca91b:0x1834091726bd0890539db04a22b673a6d9262d8b350dffaefdb10abc72e2113d}, Lyra2NonceTransitionBlock:<nil>, HIPVeldinFBlock:500009}
Receipts:
  0: cumulative: 1326462 gas: 1326462 contract: 0x0376fC3D62F7a6C4e0707dE9aAb9988a708Be183 status: 0 tx: 0x15d76766b05cbd2ce01b23fa6469c412ee0ce093d8594081e80a44d0cec7ab7d logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state:
##############################

ERROR[02-16|13:47:25.796] Failed to retrieve beacon bounds for bad block reporting err="beacon sync not yet started"

https://explorer.hypra.network/block/1600994/transactions

@JABirchall
Copy link
Author

JABirchall commented Feb 16, 2024

Ok, i cant figure out what is or isnt enabled for the Gaspar fork to work.

Gaspar was just to upgrade the EVM on Hypra from Berlin to Shanghai, which as far as i can see is EIP3855, EIP3860, EIP3198, EIP3529.

@JABirchall
Copy link
Author

JABirchall commented Feb 19, 2024

The issue might be the IntrinsicGas function, as in our geth this is not enabled as part of enabling the EIP3860 EVM upgrade to CREATE and CREATE2 gas

https://github.com/etclabscore/core-geth/blob/master/core/state_transition.go#L103-L109

Looks like this EIP effects more than was activated on Hypra network.

NOTE: This still doesn't sync past gaspar :(
@meowsbits
Copy link
Member

Error: invalid bloom

... to me this suggests its not the gas (since that was ==), but the receipts or their transactions..

@JABirchall
Copy link
Author

JABirchall commented Feb 21, 2024

Error: invalid bloom

... to me this suggests its not the gas (since that was ==), but the receipts or their transactions..

* https://github.com/etclabscore/core-geth/blob/master/core/block_validator.go#L113-L123

* https://eips.ethereum.org/EIPS/eip-658 ?

Would the bloom be different because the transcation failed? As that transcation used 100% gas which succeeds on mainnet, but if the there is a gas difference. between whats expected and what core-geth gives, would give different bloom since the gas limit was reached so it cant report gas difference?

Another indicator of this is that the local reported state has empty logs, but that transcation should have 1 log [0x8be0079c] OwnershipTransferred(),
https://explorer.hypra.network/tx/0x15d76766b05cbd2ce01b23fa6469c412ee0ce093d8594081e80a44d0cec7ab7d/logs

Do you have discord so we can have a call about it? Because im stuck, i have ideas, but testing some of them didnt change anything or completely invalidated the chain. I think doing a pair code to just get this over the line and syncing past Gaspar fork would be useful.

@meowsbits
Copy link
Member

meowsbits commented Feb 21, 2024

The issue might be the IntrinsicGas function, as in our geth this is not enabled as part of enabling the EIP3860 EVM upgrade to CREATE and CREATE2 gas [...]
Looks like this EIP effects more than was activated on Hypra network.

Can you say more about this? I went to find your source here but they look identical to me.

Have you tried comparing results of trace_call on both clients?

I'm willing to meet, but would like a little more context about what you've tried and haven't yet.

@JABirchall
Copy link
Author

JABirchall commented Feb 21, 2024

Can you say more about this? I went to find your source here but they look identical to me.

That paramter is bound to Shanghai but we didnt activate shanghai we only activate the shanhai EVM code with our gaspar fork here and here. I missed that when doing Gaspar.

Have you tried comparing results of trace_call on both clients?

I havnt yet, not really sure how to since it rejects the block.

I'm willing to meet, but would like a little more context about what you've tried and haven't yet.

What ive tested is commited. The last thing im going to test is just defining gaspar in params and adding that to enable the evm opcodes. if thaty doesnt work im completely lost.

@JABirchall
Copy link
Author

Ok, I fixed the EVM issue it looks like its now fully syncing.

Currently doing a full sync from start, will see if it keeps insync.

@JABirchall
Copy link
Author

JABirchall commented Mar 8, 2024

Ok, its still syncing, but very slowly.

There seems to be a performance issue with pebble, switching to leveldb and syncing is fast again.

Full import completed with no invalid blocks

✅ Status Syncs mainnet

Just need to figure out why its trying to generate the same dag multiple times and delete it multiple time. It was really bad on epoch 5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants