Spray JSON

So you’re making yet another Scala app and you need to parse or write some data to file/database/webservice.

What do you do? If you’re not already inside an (awesome) framework like Play Framework you will probably search “Scala JSON” and find something like Spray JSON. That’s what I did. So then you write:

import scalax.io._
import spray.json._
import DefaultJsonProtocol._

val jsonString = largeMapObject.toJson.compactPrint
Resource.fromFile("someFile").write(jsonString)(Codec.UTF8)

… and you get an exception upon either writing, or parsing this string after it was only partially written. You have several megabytes of this so just writing to a String will not work.

You will need a JSON printer that can output streams or handle writers. After searching I found Scala Stuff’s json-parser that has support for several JSON AST libraries like Spray. With json-parser writing large JSON objects is just a matter of:

import org.scalastuff.json.spray._
val writer = new PrintWriter(new File(file))
new SprayJsonPrinter(writer, 0)(largeMapObject.toJson)
writer.close()

and reading is just as easy:

val reader = Source.fromFile(file).bufferedReader()
val jsValue = SprayJsonParser.parse(reader)

Although this basic usage of Spray and Scala Stuff’s json-parser allow you to parse and write large JSON object, your machines memory is still a bottleneck as they keep the resulting JSON AST in memory. When dealing with huge JSON streams I recommend that you take a look at json-parser’s JsonHandler trait or Play Frameworks JSON Transformers.