The code looks good. What error are you getting?

Try replacing stream2 %Stream.FileCharacter with %Stream.FileBinary.

Try replacing stream1 %Stream.FileCharacter with %Stream.TmpBinary.

If you have message sample less than 3,5 mb in size try to write a test without intermediate stream.

You probably should write to a pdf file and not a txt one.

If you can, get a sample original/decoded file. Compare original file and your stream2 file using hex editor to spot differences.

I try to separate SQL from ObjectScript code. So mainly, I'm using queries, for example:

/// Some report. To display in terminal call:
/// do ##class(class).reportFunc().%Display()
Query report(date As %String = {$zd($h-1,8)}) As %SQLQuery(SELECTMODE = "ODBC")
{
SELECT
  ID,
  Value,
  EventDate
FROM myTable
WHERE EventDate>=TO_POSIXTIME(:date,'YYYYMMDD') AND EventDate<TO_POSIXTIME(:date+1,'YYYYMMDD')
ORDER BY EventDate
}

Queries can be calles from ObjectScript using autogenerated Func method:

/// Really %sqlcq.<NAMESPACE>.cls<NUMBER>
#dim rs As %SQL.ISelectResult
set rs = ..reportFunc(date)
//do rs.%Display()

while rs.%Next() {
    write rs.ID,!
}

I found this approach improves readability of the codebase. More about queries in this article.

Special case - one value.

Sometimes you don't need a resultset, but one value. In that case:

If you know ID it's possible to use GetStored method:

set value = ##class(test).<PropertyName>GetStored(ID)

If you know unique indexed value but don't know ID, it's possible to get id with Exists method:

ClassMethod <IndexName>Exists(val, Output id) As %Boolean

And after that use GetStored method.

More on auto-generated methods, such as GetStored and Exists in this article.

Finally, if you can't use above methods or you need one value but it's an aggregate, use embedded SQL if it's a short SQL and Query if it's long.

I' do it in 2 steps.

  1. Iterate over data array and build a temp local structure to hold all additional items you need
  2. Iterate over this new structure and add these items to data array.

You can add to array using %Push method:

do Obj.data.%Push(newItem)

If you want to push at a specific posiiton, use %Set

do Obj.data.%Set(position, newItem)

That said your later structure contains data which does not exist in the original structure (text values for projects and sub-projects) so you need to get it from somewhere.

Also is project - subproject hierarchy one-level, or there could be an arbitrary number of sub-project levels (i.e. 10  10-1 → 10-1-1 → 10-1-1-1)?

Okay, even if the classes/globals are the same there's a solution. Let's say you have Sample.Person class in namespaces SAMPLES and USER, each with their own data:

Class Sample.Person Extends %Persistent
{

Property Name;

Storage Default
{
<Data name="PersonDefaultData">
<Value name="1">
<Value>%%CLASSNAME</Value>
</Value>
<Value name="2">
<Value>Name</Value>
</Value>
</Data>
<DataLocation>^Sample.PersonD</DataLocation>
<DefaultData>PersonDefaultData</DefaultData>
<IdLocation>^Sample.PersonD</IdLocation>
<IndexLocation>^Sample.PersonI</IndexLocation>
<StreamLocation>^Sample.PersonS</StreamLocation>
<Type>%Library.CacheStorage</Type>
}

}


And you want to query both from the USER namespace. In that case create a new class extending Sample.Person in the USER namespace and modify storage like this:

Class Utils.SamplesPerson Extends (%Persistent, Sample.Person)
{

Storage Default
{
<Data name="PersonDefaultData">
<Value name="1">
<Value>%%CLASSNAME</Value>
</Value>
<Value name="2">
<Value>Name</Value>
</Value>
</Data>
<DataLocation>^["SAMPLES"]Sample.PersonD</DataLocation>
<DefaultData>PersonDefaultData</DefaultData>
<ExtentSize>200</ExtentSize>
<IdLocation>^["SAMPLES"]Sample.PersonD</IdLocation>
<IndexLocation>^["SAMPLES"]Sample.PersonI</IndexLocation>
<StreamLocation>^["SAMPLES"]Sample.PersonS</StreamLocation>
<Type>%Library.CacheStorage</Type>
}

Note that DataLocation, IndexLocation, IdLocation and StreamLocation point to the SAMPLES namespace.

Now query:

SELECT *
FROM Utils.SamplePerson

Would fetch data from SAMPLES namespace.

Some ideas.

1. The problem is isc.rabbitmq.API  class was imported with an error. Delete this class and try this code (post output):

Set class = "isc.rabbitmq.API"
Set classPath = ##class(%ListOfDataTypes).%New()
Do classPath.Insert(PATH-TO-JAR)
Set gateway = ##class(%Net.Remote.Gateway).%New() 
Set sc = gateway.%Connect("localhost", PORT, $Namespace, 2, classPath)
Zwrite sc
Set sc = gateway.%Import(class)
Zwrite sc
Write ##class(%Dictionary.CompiledClass).%ExistsId(class)

2. Try recompiling isc.rabbitmq.API  class.

3. Maybe have 2 amqp jars with com.rabbitmq.tools.jsonrpc.JsonRpcException class.  The only class you should import is isc.rabbitmq.API  class. It shouldn't pull many additional ones.

4. What version of AMQP are you using? I'm using amqp-5.0.0. Try the same verison?

Abstract class could have this method:

ClassMethod GetNew() As test.Order {
  set class = "test.Order"
  if ..GetCompileDate($classname())'=..GetCompileDate(class) {
    set sc = $system.OBJ.Compile($classname(),"cukb /display=none")
  }
  quit $classmethod(class, "%New")
}

ClassMethod GetCompileDate(class As %Dictionary.CacheClassname) [CodeMode = expression]
{
$$$comClassKeyGet(class,$$$cCLASStimechanged)
}

Eduard, that's a good suggestion, but one issue I see is that if you are processing millions of documents, that's a lot of database overhead. Using a simple global reference with locks would work better as you would only have n files persisted up to the Pool Size. 

The solution I outlined at all times contains exactly up to PoolSize records, so I don't think it's a very big overhead. You can lock filenames I suppose, why not?

My next challenge is to determine a better way for the file service to pick off files from the OS folder without loading all the files into a potentially massive result set which can blow out local memory and in extreme cases quickly blow out CacheTemp DB size.

Ensemble File inbound adapter uses FileSet from %File class. This query uses $zsearch to iterate over files in a directory and populates a ppg with results at once. Calling Next on that result set only moves the ppg key. You can rewrite FileSet query to advance $zsearch  on a Next call. Don't know how it would affect performance though.

I suppose you can solve this problem by separating your persistent class into abstract definition and persistent "storage only" class.

It could look like this:

Class test.Abstract.Order [ Abstract ]

{

Property a;

Method doStuff()

{

}

}

and persistent class:

Class test.Order Extends (test.Abstract.Order, %Persistent)

{

/*Generated Storage */

}

And only map test.Abstract package. This way you'll need to:

  • Copy test.Order class manually (but only once, as it does not change).
  • Recompile test.Order class in each namespace if Abstract realization changes.

But in this setup persistent class could be tuned, etc.. Also you can automate deployment steps with CI/CD tools.

What does this command return?

dumpbin /headers c:/InterSystems/IRIS_2019/CSP/bin/CSPa24.dll

For me (on IRIS for Windows (x86-64) 2018.2 (Build 241U) Fri Jan 26 2018 01:06:37 EST) it shows:

Dump of file ./CSPa24.dll

PE signature found

File Type: DLL

FILE HEADER VALUES
             14C machine (x86)
               5 number of sections
        5BC558F7 time date stamp Tue Oct 16 06:20:23 2018
               0 file pointer to symbol table
               0 number of symbols
              E0 size of optional header
            2102 characteristics
                   Executable
                   32 bit word machine
                   DLL

Note 14C machine (x86) value, it means that it's a 32bit dll. If you have similar output, I recommend installing 32 bit apache.

I have not tested the code on Windows, but here's my idea.

As you can see in the code for test method in cause of exceptions I end all my processes with

do $system.Process.Terminate(, 1)

it seems this path is getting hit.

How to fix this exception:

  1. Check that test method actually gets called. Write to a global in a first line.
  2. In exception handler add do ex.Log() and check application error log to see exception details.