robur/opam-mirror
4
0
Fork 0
Code Issues 6 Pull requests 2 Projects Releases Packages Wiki Activity Actions

Reserve an area for dumping the Serve.t -- basically the tarball and last_modified date #28

Open
hannes wants to merge 2 commits from index-startup into main
pull from: index-startup
merge into: robur:main
Conversation 2 Commits 2 Files changed 2 +163 -70
hannes commented 2024-11-21 13:21:02 +00:00
Owner
Copy link

After each tarball creation to dump the index

Also, revise startup:

  • read index, if successful, start HTTP server
    • schedule an initialization of the git_kv later (and potential dump)
    • schedule disk check later
  • if not successful
    • initialize git_kv (may need to clone opam repository)
    • potentially schedule dump
    • schedule disk check

This basically avoids the slow start since it reads the git in the background in the happy case. See #20 (and #18).

/cc @reynir

After each tarball creation to dump the index Also, revise startup: - read index, if successful, start HTTP server - schedule an initialization of the git_kv later (and potential dump) - schedule disk check later - if not successful - initialize git_kv (may need to clone opam repository) - potentially schedule dump - schedule disk check This basically avoids the slow start since it reads the git in the background in the happy case. See #20 (and #18). /cc @reynir
hannes added 1 commit 2024-11-21 13:21:03 +00:00 
After each tarball creation to dump the index

Also, revise startup:
- read index, if successful, start HTTP server
  - schedule an initialization of the git_kv later (and potential dump)
  - schedule disk check later
- if not successful
  - initialize git_kv (may need to clone opam repository)
  - potentially schedule dump
  - schedule disk check
reynir reviewed 2024-11-21 17:52:08 +00:00
reynir left a comment
Owner
Copy link

You don't zero out the index partition when initializing the disk. I will suggest a change (possibly in another PR) that makes it take an optional argument: --initialize-disk=[soft|hard] where one will just update the GPT table while the other resets everything. Or something like that.

I will do a closer review on a local checkout of the code. But it looks promising to me!

You don't zero out the index partition when initializing the disk. I will suggest a change (possibly in another PR) that makes it take an optional argument: `--initialize-disk=[soft|hard]` where one will just update the GPT table while the other resets everything. Or something like that. I will do a closer review on a local checkout of the code. But it looks promising to me!
mirage/unikernel.ml Outdated
@ -718,6 +723,46 @@ stamp: %S
mutable index : string ;
}
let to_string { commit_id ; modified ; repo ; index } =
reynir commented 2024-11-21 17:33:47 +00:00
Owner
Copy link

We can also use Marshal like we do for checksum caches. It would make of_string easier, I think.

We can also use Marshal like we do for checksum caches. It would make `of_string` easier, I think.
hannes commented 2024-11-22 11:24:59 +00:00
Author
Owner
Copy link

that's a good suggestion. yes we should :) (I noticed we already use Marshal for the md5/sha512 maps)

that's a good suggestion. yes we should :) (I noticed we already use Marshal for the md5/sha512 maps)
hannes commented 2024-11-22 11:28:13 +00:00
Author
Owner
Copy link

done in 4ae8486

done in 4ae8486
hannes marked this conversation as resolved
mirage/unikernel.ml
@ -1120,0 +1177,4 @@
~error_handler:(fun _ ?request:_ _ _ -> ())
(Serve.dispatch serve disk (K.hook_url ()) (update serve))
in
let `Initialized th = Paf.serve service t in
reynir commented 2024-11-21 17:45:51 +00:00
Owner
Copy link

Shouldn't this be let (`Initialized th) = ... in?

Shouldn't this be ``let (`Initialized th) = ... in``?
reynir commented 2024-11-21 18:05:45 +00:00
Owner
Copy link

Oh, apparently not! It compiles fine. It probably changed at the same time when you can write match _ with Ok Data data -> ...`

Oh, apparently not! It compiles fine. It probably changed at the same time when you can write `match _ with Ok `Data data -> ...`
reynir marked this conversation as resolved
reynir commented 2024-11-21 18:21:06 +00:00
Owner
Copy link

I tried to test this, but I get the following error:

Long log output 
2024-11-21T18:14:19-00:00: [DEBUG] [mimic] Try to instantiate <tls>.
2024-11-21T18:14:19-00:00: [DEBUG] [happy-eyeballs] [1] connect: id 1 host github.com
2024-11-21T18:14:19-00:00: [DEBUG] [happy-eyeballs] [1] actions: resolve AAAA github.com
resolve A github.com
2024-11-21T18:14:19-00:00: [DEBUG] [happy-eyeballs.mirage] [1] action resolve AAAA github.com
2024-11-21T18:14:19-00:00: [DEBUG] [happy-eyeballs] [1] connect_ip id 2 dsts 2001:67c:28a4:::853, 91.239.100.100:853
2024-11-21T18:14:19-00:00: [DEBUG] [happy-eyeballs] [1] actions: 2 connect 2001:67c:28a4:: (using 2001:67c:28a4:::853), attempt 0
2024-11-21T18:14:19-00:00: [DEBUG] [happy-eyeballs.mirage] [1] action 2 connect 2001:67c:28a4:: (using 2001:67c:28a4:::853), attempt 0
2024-11-21T18:14:19-00:00: [WARNING] [tcpip-stack-direct] attempted to write an IPv6 packet in a v4 only stack
2024-11-21T18:14:19-00:00: [DEBUG] [happy-eyeballs.mirage] [1] action resolve A github.com
2024-11-21T18:14:19-00:00: [DEBUG] [happy-eyeballs] [1] timer continue true, 1 actions: 2 connect 2001:67c:28a4:: (using 91.239.100.100:853), attempt 1
2024-11-21T18:14:19-00:00: [DEBUG] [happy-eyeballs.mirage] [1] action 2 connect 2001:67c:28a4:: (using 91.239.100.100:853), attempt 1
2024-11-21T18:14:20-00:00: [DEBUG] [happy-eyeballs] [1] timer continue true, 2 actions: resolve A github.com
resolve AAAA github.com
2024-11-21T18:14:20-00:00: [DEBUG] [happy-eyeballs.mirage] [1] action resolve A github.com
2024-11-21T18:14:20-00:00: [DEBUG] [happy-eyeballs.mirage] [1] action resolve AAAA github.com
2024-11-21T18:14:20-00:00: [DEBUG] [happy-eyeballs] [1] timer continue true, 2 actions: resolve A github.com
resolve AAAA github.com
2024-11-21T18:14:20-00:00: [DEBUG] [happy-eyeballs.mirage] [1] action resolve A github.com
2024-11-21T18:14:20-00:00: [DEBUG] [happy-eyeballs.mirage] [1] action resolve AAAA github.com
2024-11-21T18:14:20-00:00: [DEBUG] [happy-eyeballs] [1] timer continue true, 1 actions: 1 connect failed github.com: timeout
2024-11-21T18:14:20-00:00: [DEBUG] [happy-eyeballs.mirage] [1] action 1 connect failed github.com: timeout
2024-11-21T18:14:20-00:00: [DEBUG] [happy-eyeballs.mirage] [1] connection failed to github.com after 1.026s
2024-11-21T18:14:20-00:00: [DEBUG] [git.sync] Load locals references.
2024-11-21T18:14:20-00:00: [DEBUG] [git.mem] Ref.list.
2024-11-21T18:14:20-00:00: [DEBUG] [smart_flow] Start to read 65536 byte(s).
2024-11-21T18:14:20-00:00: [ERROR] [smart_flow] Got an error: Handshake got an error.
2024-11-21T18:14:20-00:00: [DEBUG] [git-fetch] End of pack.
Fatal error: exception Invalid_argument("error fetching: Handshake got an error")
Raised at Stdlib.invalid_arg in file "stdlib.ml", line 30, characters 20-45
Called from Git_kv.connect.(fun) in file "duniverse/git-kv/src/git_kv.ml", line 107, characters 11-23
Called from Lwt.Sequential_composition.bind.create_result_promise_and_callback_if_deferred.callback in file "duniverse/lwt/src/core/lwt.ml", line 1844, characters 16-19
Re-raised at Lwt.Miscellaneous.poll in file "duniverse/lwt/src/core/lwt.ml", line 3123, characters 20-29
Called from Solo5_os__Main.run.aux in file "duniverse/mirage-solo5/lib/main.ml", line 46, characters 10-20
Called from Dune__exe__Main.run in file "mirage/main.ml" (inlined), line 4, characters 12-31
Called from Dune__exe__Main in file "mirage/main.ml", line 599, characters 2-7
Solo5: solo5_exit(2) called

It seems happy-eyeballs fails to connect, but ocaml-git is entirely unhelpful in explaining what went wrong.

It's entirely possible the network setup is broken.

I tried to test this, but I get the following error: <details> <summary>Long log output</summary> ``` 2024-11-21T18:14:19-00:00: [DEBUG] [mimic] Try to instantiate <tls>. 2024-11-21T18:14:19-00:00: [DEBUG] [happy-eyeballs] [1] connect: id 1 host github.com 2024-11-21T18:14:19-00:00: [DEBUG] [happy-eyeballs] [1] actions: resolve AAAA github.com resolve A github.com 2024-11-21T18:14:19-00:00: [DEBUG] [happy-eyeballs.mirage] [1] action resolve AAAA github.com 2024-11-21T18:14:19-00:00: [DEBUG] [happy-eyeballs] [1] connect_ip id 2 dsts 2001:67c:28a4:::853, 91.239.100.100:853 2024-11-21T18:14:19-00:00: [DEBUG] [happy-eyeballs] [1] actions: 2 connect 2001:67c:28a4:: (using 2001:67c:28a4:::853), attempt 0 2024-11-21T18:14:19-00:00: [DEBUG] [happy-eyeballs.mirage] [1] action 2 connect 2001:67c:28a4:: (using 2001:67c:28a4:::853), attempt 0 2024-11-21T18:14:19-00:00: [WARNING] [tcpip-stack-direct] attempted to write an IPv6 packet in a v4 only stack 2024-11-21T18:14:19-00:00: [DEBUG] [happy-eyeballs.mirage] [1] action resolve A github.com 2024-11-21T18:14:19-00:00: [DEBUG] [happy-eyeballs] [1] timer continue true, 1 actions: 2 connect 2001:67c:28a4:: (using 91.239.100.100:853), attempt 1 2024-11-21T18:14:19-00:00: [DEBUG] [happy-eyeballs.mirage] [1] action 2 connect 2001:67c:28a4:: (using 91.239.100.100:853), attempt 1 2024-11-21T18:14:20-00:00: [DEBUG] [happy-eyeballs] [1] timer continue true, 2 actions: resolve A github.com resolve AAAA github.com 2024-11-21T18:14:20-00:00: [DEBUG] [happy-eyeballs.mirage] [1] action resolve A github.com 2024-11-21T18:14:20-00:00: [DEBUG] [happy-eyeballs.mirage] [1] action resolve AAAA github.com 2024-11-21T18:14:20-00:00: [DEBUG] [happy-eyeballs] [1] timer continue true, 2 actions: resolve A github.com resolve AAAA github.com 2024-11-21T18:14:20-00:00: [DEBUG] [happy-eyeballs.mirage] [1] action resolve A github.com 2024-11-21T18:14:20-00:00: [DEBUG] [happy-eyeballs.mirage] [1] action resolve AAAA github.com 2024-11-21T18:14:20-00:00: [DEBUG] [happy-eyeballs] [1] timer continue true, 1 actions: 1 connect failed github.com: timeout 2024-11-21T18:14:20-00:00: [DEBUG] [happy-eyeballs.mirage] [1] action 1 connect failed github.com: timeout 2024-11-21T18:14:20-00:00: [DEBUG] [happy-eyeballs.mirage] [1] connection failed to github.com after 1.026s 2024-11-21T18:14:20-00:00: [DEBUG] [git.sync] Load locals references. 2024-11-21T18:14:20-00:00: [DEBUG] [git.mem] Ref.list. 2024-11-21T18:14:20-00:00: [DEBUG] [smart_flow] Start to read 65536 byte(s). 2024-11-21T18:14:20-00:00: [ERROR] [smart_flow] Got an error: Handshake got an error. 2024-11-21T18:14:20-00:00: [DEBUG] [git-fetch] End of pack. Fatal error: exception Invalid_argument("error fetching: Handshake got an error") Raised at Stdlib.invalid_arg in file "stdlib.ml", line 30, characters 20-45 Called from Git_kv.connect.(fun) in file "duniverse/git-kv/src/git_kv.ml", line 107, characters 11-23 Called from Lwt.Sequential_composition.bind.create_result_promise_and_callback_if_deferred.callback in file "duniverse/lwt/src/core/lwt.ml", line 1844, characters 16-19 Re-raised at Lwt.Miscellaneous.poll in file "duniverse/lwt/src/core/lwt.ml", line 3123, characters 20-29 Called from Solo5_os__Main.run.aux in file "duniverse/mirage-solo5/lib/main.ml", line 46, characters 10-20 Called from Dune__exe__Main.run in file "mirage/main.ml" (inlined), line 4, characters 12-31 Called from Dune__exe__Main in file "mirage/main.ml", line 599, characters 2-7 Solo5: solo5_exit(2) called ``` </details> It seems happy-eyeballs fails to connect, but ocaml-git is entirely unhelpful in explaining what went wrong. It's entirely possible the network setup is broken.
reynir commented 2024-11-21 19:40:35 +00:00
Owner
Copy link

Likely my setup is broken /o\ with a different version of opam-mirror I get the same error.

Likely my setup is broken /o\ with a different version of opam-mirror I get the same error.
hannes added 1 commit 2024-11-22 11:27:57 +00:00
This pull request has changes conflicting with the target branch.
  • mirage/unikernel.ml
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin index-startup:index-startup
git checkout index-startup

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git checkout main
git merge --no-ff index-startup
git checkout index-startup
git rebase main
git checkout main
git merge --ff-only index-startup
git checkout index-startup
git rebase main
git checkout main
git merge --no-ff index-startup
git checkout main
git merge --squash index-startup
git checkout main
git merge --ff-only index-startup
git checkout main
git merge index-startup
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: robur/opam-mirror#28
No description provided.
Delete branch "index-startup"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?