There are several approaches available:

1. Automatic conversion of XSL(X) to CSV is possible using LibreOffice. Here's an article on how to do that.

2. Use Java library, such as Apache POI to traverse the cells and send results back. To send results back quickly and efficiently you can use String[] type and fill it with $lb built on Java side. String[] would become %ListOfDataTypes in Cache and $lb would remain $lb.

I've done some preliminary work but didn't get around to finishing or publishing it so here's a sample code that outputs list to console:

package isc.poi;

import org.apache.poi.hssf.usermodel.HSSFRow;
import org.apache.poi.ss.usermodel.*;
import org.apache.poi.ss.usermodel.Row;

import java.util.ArrayList;
import java.util.Iterator;
import java.io.File;

import static org.apache.poi.ss.usermodel.CellType.*;

public class Main {

    public static String ROWSEPARATOR = "\t\t\t";

    public static void main(String[] args) {
        try {
            Test1();
        } catch (Exception ex) {
        }
    }

    public static String[] Test1 () throws Exception{
        ArrayList<String> list = new ArrayList<String>();

        File file = GetFile();
        Workbook workbook = WorkbookFactory.create(file);
        Iterator<Sheet> sheetIterator = workbook.sheetIterator();
        while(sheetIterator.hasNext()){
            Sheet sheet = sheetIterator.next();
            String name  = sheet.getSheetName();
            String value = null;

            Iterator rows = sheet.rowIterator();
            while (rows.hasNext()) {
                Row row = (Row) rows.next();

                for(int i=0; i<row.getLastCellNum(); i++) {
                    Cell cell = row.getCell(i, Row.MissingCellPolicy.CREATE_NULL_AS_BLANK);
                    if (cell.getCellTypeEnum() == FORMULA) {
                        switch(cell.getCachedFormulaResultTypeEnum()) {
                            case NUMERIC:
                                value = String.valueOf(cell.getNumericCellValue());
                                break;
                            case STRING:
                                value = cell.getRichStringCellValue().getString();
                                break;
                        }
                    } else {
                        value = cell.toString();
                    }
                    list.add(value);
                    ///System.out.print("'" + cell.toString() + "'"+" ");
                }
                list.add(ROWSEPARATOR);
                System.out.println();
            }


            /*for (Row row : sheet) {
                for (Cell cell : row) {
                    System.out.print(cell.toString()+" ");
                    //int i=1;
                }
                System.out.println();
            }*/
        }
        String[] result = list.toArray(new String[list.size()]);
        return result;
    }

    public static File GetFile () {
        File file = new File("D:\\Cache\\POI\\Book1.xlsx");

        return file;
    }

    public static Object Test(Object in)
    {
        String[] ret = new String[1];
        ret[0] = "144";
        return ret;
    }
}

I'd run a BS with SQL inbound adapter called every 86400 seconds. Query is some SELECT Count(*) ...

To get optimal WHERE condition you'll probably need to answer these questions:

  • Are records immutable?
  • Can they be updated?
  • Can records be deleted?
  • Can records be inserted for a past/future dates?

If it's a local table you can run BS without adapter and just use embedded SQL.

There are two use cases here:

  • File is tied to a specific object (for example you have "Document" class and it has "scan" file). In that case you can use %FileBinaryStream property - as before getting the file you would probably open "Document" object first
  • File is not tied to a specific object. In that case you can create a separate table "Files" that stores
    • link to file as a FileBinaryStream
    • hash
    • displayed file name
    • file path
    • user who uploaded the file
    • extension
    • size
    • any other attributes you need

It would always work faster compared to OS search.

Other notes:

  • Files are immutable - if you're building an application where user can edit files, it's usually preferable to have immutable "files" objects and just create new file versions.
  • File size limits - always define and check for maximum size.
  • Extension limits - limit extensions user can upload.
  • Storage - if it's a low volume inserts (<1000/day) store files in a folder = date, otherwise generate a new folder for each new thousand of files. These approaches can be combined: date/1, date/2 ...
  • Hash name - I often store files where their OS name is their hash. This way I can quickly validate that file is valid and also it  solves the problems with non-latin characters.
  • Never store files under names supplied by user. Acceptable filenames are: guid, hash (integer id should also be avoided).
  • GZIP - in some cases using GZIP streams can save on space, especially if it's a text file. For example XML envelopes and such.

Try to check who calls GetBatchHeader and what it returns.

At a glance, I think it's called from GetBatchObject method of EnsLib.RecordMap.Service.BatchStandard class:

Set tStatus = $classmethod(..BatchClass, "GetBatchHeader", pIOStream, pTimeout, .pBatch, .pLookAhead)
If $$$ISERR(tStatus) Quit

Check that tStatus is not an error, fix GetBatchHeader it if it's an error.

You can determine that in run time with:

  • $system.CLS.GetModified(oref) to ger if oref was modified
  • m%property to get if property changed

Here's a simple example of how it all works

 Class User.Person Extends (%Persistent)
{

Property Name As %String(DISPLAYNAME = "Name");

Property Age As %Integer(DISPLAYNAME = "Age");

/// Create one user
ClassMethod Recreate() As %Status
{
  do ..%KillExtent()
  set person = ..%New()
  set person.Age = $random(100)
  set person.Name = $random(100) 
  quit person.%Save()
}

/// do ##class(User.Person).Test()
ClassMethod Test()
{
  $$$QuitOnError(..Recreate())

  set person = ..%OpenId(1)
  do person.ModificationState()

  set person.Age = $random(100)
  do person.ModificationState()

  set person.Name = $random(100)
  do person.ModificationState()
}

/// Show current object state
Method ModificationState()
{
  write "Object modified: ", $system.CLS.GetModified(), !
  write "Name prop modified: ", ..IsNameModified(), !, !
}

/// Get name property modification status
Method IsNameModified() As %Boolean
{
  quit m%Name
}

}

Executing in terminal:

>do ##class(User.Person).Test()
Object modified: 0
Name prop modified: 0
 
Object modified: 1
Name prop modified: 0
 
Object modified: 1
Name prop modified: 1

Note that changing value back would still mark property as modified.

Installer manifest can call arbitrary code.

  1. Mirrors can be managed via  ^MIRROR utility.
  2. Can be manipulated via Config.Journal class.
  3. You can configure automatic purging task and set a number of days to keep there. Docs.
  4. Tasks can be created and modified programmatically. Sample. They could also be exported and imported. Community discussion.

It's a url not a file so you need to use OpenURL for that:

set reader=##class(%XML.Reader).%New()
set httprequest=##class(%Net.HttpRequest).%New()
set httprequest.SSLConfiguration="...."
set status=reader.OpenURL("https://.....",,httprequest)

But as @Robert.Cemper pointed out it's a dtd file and so would not be parsed by XML reader.

If you want to generate classes automatically from XML schema, you'll need XSD files. Documentation.

What's your current locale? If it's not Chinese you may want to change it. Note that it significantly affects system behavior and you should understand the implications in such a change.

???? often indicates the data loss during encoding conversions consider this Cyrillic  example:

set t = "Привет"
 
USER>zzdump t
 
0000: 041F 0440 0438 0432 0435 0442                           Привет
USER>zzdump $zcvt(t, "I", "UTF8")
 
0000: 3F 3F 3F 3F 3F 3F                                       ??????
USER>zzdump $zcvt(t, "O", "UTF8")
 
0000: D0 9F D1 80 D0 B8 D0 B2 D0 B5 D1 82                     Ð.Ñ.ивеÑ.

Note the difference between question marks and the last output. Last output is just encoding translation, but the first conversion - into questions completely lost the information, so you need to look deeper.

You can try to modify TCP/IP IO table via ^NLS utility in %SYS namespace

do ^NLS
 
 
1) Display current locale
2) Select defaults
3) Change locale
4) Display loaded settings
5) Advanced

NLS option? 2
 
1) Internal tables
2) I/O tables
3) CSP files
4) Date, time and number formats
 
Category of defaults? 2
 
Items marked with (*) represent the locale's original default
 
I/O table              Current default
---------------------  --------------------
 
1) Process             RAW (*)
2) Cache Terminal      UTF8 (*)
3) Other terminal      UTF8 (*)
4) File                UTF8 (*)
5) Magtape             UTF8 (*)
6) TCP/IP              RAW (*)
7) System call         RAW (*)
8) Printer             CP1251 (*)
 
I/O table: 6
 
 1) RAW (*)
 2) UTF8
 3) UnicodeLittle
 4) UnicodeBig
 5) CP1250
 6) CP1251
 7) CP1252
 8) CP1253
 9) CP1255
10) CP866
11) CP874
12) EBCDIC
13) KOI8R
14) Latin2
15) Latin9
16) LatinC
17) LatinG
18) LatinH
19) LatinT
 
Selection for TCP/IP: 1 =>

You can create a separate query table with fields: query, user, time, arg1. arg2. ar3, ..., argN.

And log information about the queries as is. This way you would be able to:

  • group by query - to get how often some specific procedure is called regardless of the argument values
  • filter by argument values

Additionally there's ODBC logging and JDBC logging might be available depending on the database vendor.