Drop owee dependency and work on strings

reynir commented

2025-01-30 09:50:00 +00:00

Owner

Fixes #1, #2.

reynir added 1 commit 2025-01-30 09:50:01 +00:00

Drop owee dependency and work on strings 0a04cc0bcc

reynir commented

2025-01-30 09:53:37 +00:00

Author

Owner

This has the downside that the whole binary has to reside in memory as a string. I think it will be interesting to look into using cachet here.

reynir added 1 commit 2025-01-30 11:40:02 +00:00

Switch to cachet 7bef25f19f

reynir added 1 commit 2025-01-30 11:43:24 +00:00

osolo5-elftool: close file bf003875b9

reynir added 1 commit 2025-01-30 12:33:35 +00:00

Get rid of cstruct 04cc6350b1

And update opam file

reynir changed title from ~~WIP: Drop owee dependency and work on strings~~ to Drop owee dependency and work on strings

2025-01-30 12:34:20 +00:00

reynir commented

2025-01-30 12:36:42 +00:00

Author

Owner

This now uses cachet and doesn't depend on cstruct and uses cachet instead. This has a lot of benefits, but to me it seems much harder to use if the whole binary is in memory. I will need to experiment with cachet for that use case.

reynir added 1 commit 2025-01-30 13:44:40 +00:00

Catch exceptions in interface 742799de5b

reynir added 1 commit 2025-01-30 14:54:21 +00:00

Update dune files with dependency changes 649872703f

reynir commented

2025-01-31 16:25:41 +00:00

Author

Owner

The following snippet works for strings. In my test this performs fine. There were only 3 cache misses so not many bigarrays are created.

let map () ~pos len =
  if pos >= String.length unipi || len <= 0 then
    (Cachet.Bstr.empty :> Cachet.bigstring)
  else
    let len = min len (max 0 (String.length unipi - pos)) in
    let b : Cachet.bigstring = Bigarray.Array1.create Bigarray.char Bigarray.c_layout len in
    for i = 0 to len - 1 do
      b.{i} <- (unipi.[pos+i])
    done;
    b

Changes here breaks on 32 bit platforms (if that ever worked to begin with).

The following snippet works for strings. In my test this performs fine. There were only 3 cache misses so not many bigarrays are created. ```OCaml let map () ~pos len = if pos >= String.length unipi || len <= 0 then (Cachet.Bstr.empty :> Cachet.bigstring) else let len = min len (max 0 (String.length unipi - pos)) in let b : Cachet.bigstring = Bigarray.Array1.create Bigarray.char Bigarray.c_layout len in for i = 0 to len - 1 do b.{i} <- (unipi.[pos+i]) done; b ``` Changes here breaks on 32 bit platforms (if that ever worked to begin with).

reynir added 1 commit 2025-01-31 16:28:16 +00:00

Mark as unavailable on 32 bit platforms 8feb54d5f8

dinosaure commented

2025-01-31 16:33:46 +00:00

Owner

The following snippet works for strings. In my test this performs fine. There were only 3 cache misses so not many bigarrays are created.

it should be noted that bigarrays are not really counted in this case. Allocated from Unix.map_file, this is pure virtual memory :) - furthermore, you could map a 64 GB file and there'd be no impact in terms of memory used by the program.

> The following snippet works for strings. In my test this performs fine. There were only 3 cache misses so not many bigarrays are created. it should be noted that bigarrays are not really counted in this case. Allocated from `Unix.map_file`, this is pure virtual memory :) - furthermore, you could map a 64 GB file and there'd be no impact in terms of memory used by the program.

👍 1

reynir commented

2025-01-31 16:37:46 +00:00

Author

Owner

The following snippet works for strings. In my test this performs fine. There were only 3 cache misses so not many bigarrays are created.

it should be noted that bigarrays are not really counted in this case. Allocated from Unix.map_file, this is pure virtual memory :) - furthermore, you could map a 64 GB file and there'd be no impact in terms of memory used by the program.

sorry this was in the context where the "file" is not a file and is a string in memory. In albatross and mollymawk we expect an API that takes a string. In the case of mollymawk the string is the binary as in the POST request by the browser client.

I think this will likely change at some point when we perhaps stream the binary to a temporary file. Then cachet will be more useful :) in the meantime we need to be able to process strings.

> > The following snippet works for strings. In my test this performs fine. There were only 3 cache misses so not many bigarrays are created. > > it should be noted that bigarrays are not really counted in this case. Allocated from `Unix.map_file`, this is pure virtual memory :) - furthermore, you could map a 64 GB file and there'd be no impact in terms of memory used by the program. sorry this was in the context where the "file" is not a file and is a string in memory. In albatross and mollymawk we expect an API that takes a string. In the case of mollymawk the string is the binary as in the POST request by the browser client. I think this will likely change at some point when we perhaps stream the binary to a temporary file. Then cachet will be more useful :) in the meantime we need to be able to process strings.

reynir added 1 commit 2025-01-31 16:46:52 +00:00

Add cram test e98cc6d669

The test uses osolo5-elftool on the test_hello application from the
solo5 tests to query their abi and manifest. The manifest is empty and
boring.

reynir added 1 commit 2025-01-31 17:01:07 +00:00

Add cram dependency 9e0ecbef19

reynir added 1 commit 2025-01-31 17:09:35 +00:00

Try to make CI happy with cram test bba9fc5251

reynir force-pushed no-owee from bba9fc5251 to 9e0ecbef19

2025-01-31 17:10:42 +00:00

Compare

reynir merged commit 514fe9a060 into main

2025-01-31 17:11:26 +00:00

reynir deleted branch no-owee

2025-01-31 17:11:28 +00:00

reynir referenced this pull request from a commit

2025-01-31 17:11:29 +00:00

Merge pull request 'Drop owee dependency and work on strings' (!2) from no-owee into main

Drop owee dependency and work on strings #2