s&box

Seven

Member

JoinedMay 2023 Posts36 Score1,310

My game requires downloading a big file to disk and then parsing it on load in order to provide quick look up of object definitions and data. sbox didn't have a way of doing this without loading the entire thing into memory which I really wanted to avoid.

because the bulk data i fetch is returned as a JSON array [ {...}, {...} ] I wrote this method as apart of my file interactions wrapper class. I'm sharing it here because no one really seemed to have an idea on how do this in the way I wanted.

I really don't know what i'm doing, this is pretty far outside of what I normally code so if you have any suggestions for improving memory efficiency i'm open to them. This is just the cleanest, working version I could come with after a lot of googling and talking with AI agents to explain memory topics.

private static IEnumerable<string> StreamTopLevelArrayObjects( Stream stream, CancellationToken token )
{
    // Strategy:
    // - Read UTF8 bytes in chunks
    // - Find '[' then repeatedly extract JSON objects at top-level: {...}
    // - Track brace depth and JSON string escaping to avoid false matches
    //
    // This assumes the top-level is a JSON array and elements are objects.
    //
    // Bounded memory: holds only current object text.
    var buf = ArrayPool<byte>.Shared.Rent( BufferSize );
    try
    {
       var sb = new StringBuilder( 64 * 1024 );

       var startedArray = false;
       var inString = false;
       var escape = false;

       var capturingObject = false;
       var braceDepth = 0;

       while ( true )
       {
          token.ThrowIfCancellationRequested();

          var read = stream.Read( buf, 0, buf.Length );
          if ( read <= 0 )
             yield break;

          // Convert this chunk to chars. We accept chunk-boundary UTF8 splits by using a Decoder.
          // Use a single decoder instance to handle partial characters.
          // NOTE: We create it lazily and keep it static inside this iterator scope.
          // (No async/yield boundary issues here; this is a sync iterator.)
          var chars = Encoding.UTF8.GetChars( buf, 0, read );

          for ( int i = 0; i < chars.Length; i++ )
          {
             token.ThrowIfCancellationRequested();

             var c = chars[i];

             if ( !startedArray )
             {
                if ( c == '[' )
                   startedArray = true;
                continue;
             }

             if ( !capturingObject )
             {
                // Skip whitespace/commas until an object starts, or end array.
                if ( c == '{' )
                {
                   capturingObject = true;
                   braceDepth = 1;
                   inString = false;
                   escape = false;

                   sb.Clear();
                   sb.Append( c );
                }
                else if ( c == ']' )
                {
                   yield break;
                }

                continue;
             }

             // Capturing an object: append and update JSON string/brace tracking.
             sb.Append( c );

             if ( inString )
             {
                if ( escape )
                {
                   escape = false;
                   continue;
                }

                if ( c == '\\' )
                {
                   escape = true;
                   continue;
                }

                if ( c == '"' )
                {
                   inString = false;
                   continue;
                }

                continue;
             }

             // Not in string
             if ( c == '"' )
             {
                inString = true;
                continue;
             }

             if ( c == '{' )
             {
                braceDepth++;
                continue;
             }

             if ( c == '}' )
             {
                braceDepth--;
                if ( braceDepth == 0 )
                {
                   capturingObject = false;
                   yield return sb.ToString();
                }
             }
          }
       }
    }
    finally
    {
       ArrayPool<byte>.Shared.Return( buf );
    }
}

again I really don't have the best idea as to what i'm doing with this but it works the way I wanted (I think?) so Id thought I share incase anyone else is in a similar boat to me

8 months ago

#2

antopilo

Facepunch

JoinedFeb 2023 Posts4 Score745

Heya, we just merged a feature that might fit your needs.

We now support storing and deserialization binary resources in basically anything that gets serialized as JSON.

Check out the doc, ping me if you have any questions:
https://sbox.game/dev/doc/systems/assetsresources/binary-serialization/

8 months ago

#3

Seven

Member

JoinedMay 2023 Posts36 Score1,310

The example given on that page errors,

While untested this is a compiling version of the code snippet given. on that page.

public class MyBigData : BlobData
{
    public List<float> Data { get; set; } = [];
    
    public override void Serialize( ref Writer writer )
    {
       writer.Stream.Write( Data.Count );
        
       foreach ( var instance in Data )
       {
          writer.Stream.Write( instance );
       }
    }
    
    public override void Deserialize( ref Reader reader )
    {
       var instanceCount = reader.Stream.Read<int>();
        
       for( int i = 0; i < instanceCount; i++ )
       {
          Data.Add( reader.Stream.Read<float>() );
       }
    }
}