Questions on binary blobs

[mik1234mc]

Hi all,

I have few questions on ArcadeDB since I am in the decision process if I should choose OrientDB or ArcadeDB for my project.

1. Storing binary blobs in the ArcadeDB. Is it now supported somehow? Is there example how to store binary blobs via, e.g., HTTP API? OrientDB seems to be ready for large binary blobs with ORecordBytes (https://orientdb.com/docs/last/java/Binary-Data.html).
2. Does OrientDB importer into ArcadeDB supports also importing binary blobs stored as ORecordBytes in OrientDB?
3. Is there any activity to have also Python language drivers?

Thank you,
Michael

[lvca]

ArcadeDB doesn't support BLOB records like OrientDB. However, you can store binary data in record properties:

1. by setting a byte array as property: document.set("binary", new byte[] {})
2. as string and convert from/to binary by using the functions encode() and decode(): https://docs.arcadedb.com/#_encode

Of course, solution 1 is the most efficient. To emulate the BLOB record you could create a Blob type with only one property of binary type and use that. Example:

public class BLOBTest extends TestHelper {
  @Override
  public void beginTest() {
    database.transaction(() -> {
      DocumentType type = database.getSchema().createDocumentType("Blob");
      type.createProperty("binary", Type.BINARY);
    });
  }

  @Test
  public void testWriteAndRead() {
    database.transaction(() -> {
      final MutableDocument blob = database.newDocument("Blob");
      blob.set("binary", "This is a test".getBytes());
      blob.save();
    });

    database.transaction(() -> {
      final Document blob = database.iterateType("Blob", false).next().asDocument();
      Assertions.assertEquals("This is a test", new String( blob.getBinary("binary")));
    });
  }
}

NOTE: The example above compiles only with the latest 22.10.1-SNAPSHOT because of the new getBinary() method in Document class. To run it on older versions, just replace getBinary(<property>) with (byte[]) get(<property>)

About the other 2 questions:
2. no
3. it's already there if you use BINARY data types in documents/vertices/edges

[lvca]

Also added a new chapter about binary data: https://docs.arcadedb.com/#Binary-Types.

[mik1234mc]

Thank you @lvca.

I am just adding SQL commands which might be useful for newcomers how to use binary data over HTTP/SQL. Create BINARY property via SQL: CREATE PROPERTY Blob.binary BINARY.

First take your binary data, e.g., "light work." and generate base64 representation (you can use online tools for ASCII or HEX transformation into base64 for initial manual tests). In our case, we get "bGlnaHQgd29yay4=".

Then execute INSERT INTO Blob SET binary = decode('bGlnaHQgd29yay4=','base64') which stores the base64 in binary format. To retrieve data execute SELECT encode(binary, 'base64') From Blob. The second command returns "bGlnaHQgd29yay4=" which can be transferred again into "light work." string.

[tolgaulas]

@lvca i've seen in documentation that the BINARY type has a 2GB limit, but there is no limit mentioned in string. Can string store larger then that, for example 10GB, of base64 data? If yes; is it wise or should we look for an external storage coupling for those?

[lvca]

Sam elimitations for any type and a record in general: can't be larger than 2GB. I'm against saving large files/content into ArcadeDB. It's always more efficient to use the file system for this and save only the path to the file into ArcadeDB.

[mik1234mc]

@lvca, I agree with the approach to save the large files directly into file system while keeping only the path in ArcadeDB, but I think there should be some unified API for this. For instance, I would like to use ArcadeDB with High Availability mode, how the files stored in the file systems should be replicated among the servers?

[tolgaulas]

I think even 2GB binary is overkill for the purpose of this DB.. My two cents is if you need more then a few KB of binary data to be stored, I am assuming its something a data to be interfpreted as a file. Arcadedb can "recommend" something.. As Minio is no more for S3 store, there are a few "open source" rusty initiative have jumped in and they be friends for such case. If its modular, then any S3 (or equivalent store) can be "accoupled"