www.lim.com
Commodity DataServer Release Notes
This document details the major enhancements for each release. For a listing of all changes and bug fixes, please see the Change Log.
Note: The "CHANGES.all" file is bundled with each release.
New Features:
[MIM- 60] - 64-bit MIM C API for Windows
[MIM-168] - Added an -x option to maintain the slaves are in an "always" running state
Bug Fixes:
[MIM- 32] - Resolved MIM db locked 4091 issue
[MIM-148] - Resolved issue when C-stored function argument is incorrect
[MIM-150] - Resolved relation_column_add issue where aggregation was not explicitly set for composite column
[MIM-151] - Resolved issue where mimdbcp doesn't copy any data if a data series begins with a NaN
[MIM-162] - Resolved issue with logging web service api call
[MIM-164] - Resolved issue where cstored_function hf doesn't run on new server binaries (xmim_4.7.01.43)
[MIM-166] - Resolved Regen Continuous issue with Formulas
[MIM-171] - Resolved issue where mimdbcp did not properly handle integer NaN values
Enhancements:
[MIM-149] - Enabled the ~ within database_narrow to be interpreted as $LIMHOME
[MIM-156] - Enabled logging for api getRelChildrenWithTypes()
[MIM-167] - Improve startup of slaves from the present arbitrary sleep method
[MIM-143] - Resolved Time Series issue with holiday dates/time not presented properly when retrieved with SelectRecords
[MIM-135] - Enhancement added apis XmimVASelectRecordsDouble(), XmimSelectRecordsDouble(), and XmimTestNaNDouble() to facilitate double types as opposed to floats.
[MIM-135] - addressed an issue with XmimSelectRecords() and XmimVASelectRecords() returning an erroneous result if there are no values to return. This was also fixed in XmimSelectRecordsDouble and XmimVASelectRecordsDouble().
[MIM-140] - MIM Server coring with new 4.7.01.44 binaries
[MIM-134] - Resolved unneeded futures rollover processing in query engine which was causing errors with futures formulas
Enhancement:
[MIM-131] - Added ability to remove futures contract expiration day and first notice day
Bug Fixes:
[MIM-133] - Resolved issue wher relation_alias_add command did not overwrite target when target already exists
[MIM-130] - Resolved issue where publisher was producing multiple deletes erroneously for continuous relations
[MIM-128] - Resolved issue involving XMIM Queries that excecute for long periods of time
[MIM-129] - Resolved issue with logs being spammed by entries indicating repeated attempts to shut down idle slave servers.
[MIM-125] - Enhancement to .xmimrc option: echo “webservices: yes” to support user names containing ‘@’ symbols
[MIM-71] - Resolved NaN detection on get_records_doubles() manifesting itself in the republisher
[MIM-114] - Resolved issue with server when a formula involves regen_continuous.
[MIM-112] - Fix for memory issue with the Formula loader.
[MIM-95] Fixed the slow down of 4091 writer slave
[MIM-106] Resolved server issue that causes an error for query involving futures symbol
[MIM-107] Enhancement created to detect when a client has terminated command and checks to insure that any locks it was holding are released
[MIM-110] Changed units_measure_load memory management
[MIM-91] Changes were made to the API to add XmimGetRelChildrenWithTypes and XmimGetColChildrenWithTypes
[MIM-105] Client Closer Detection on Linux - Client terminates before calling unlock/disconnect on server
[MIM-101] Resolved Units of Measure errors to the server
changed class xmim_server method rpc_get_kids_with_types from private to public (modified api_svr_svc.H)
Fix logging format spec for MIM Server error reporting. (modified api_svr_svc.C)
Fixed an issue where deletes identified in the publisher for continuous contracts were not reflected in their corresponding continuous futures. (MIM-63 or 14764)
Enhanced the capability to store large blocks of data in the Mim Server. In earlier versions the limit for a relation size is 512, with this release and the proper flag set in the .xmimrc (bigBlockData: yes) along with a specific client library, we have increased this limit to 16K. The client library and programs (bmim_client, xmim_get, …) required to allow this to work is only available upon request and will not be distributed with this release.
Fixed an issue with the Logging. This issue caused the MIM Server to terminate unexpectedly when a specific error message was written.
Fixed an issue during roll over where roll over did not incorporate contracts that were loaded from a different update file.
Enhanced the solaris platform by better managing the lock counts and connections to the server, taking appropriate action if the connection is terminated prematurely by the client or due to a network outage.
Reduced the time the Master process starts up the slave processes
Fixed issue where the slave servers were failing in the logging mechanism.
Fixed memory leak found when using mimdbcp
Fixed an issue where rollover publishing generated invalid delete entries after holiday/weekend NaN values.
Added a ‘forced’ regen after writer timeout period. New xmim_server -z <timeout minutes> for 4091 slave process.
Added start up warning messages when C-Stored function load (or other) problems occur.
Bug Fixes:
Fixed issue where negative number were returned as NaNs. (MIM-24)
Fixed issue where real-time data did not aggregate daily in queries. (MIM-23)
Bug Fixes:
Added client handle check when logging.
Fixed time shift in queries that have attributes larger than the query’s specified execution units. (61656)
Extended logging capability to include pid, server number, and entitlements group.
Added a new log level 5 and 7.
Fixed security issue where aliases do not inherit the entitlements of the relations/columns they are referencing. (60974)
Bug Fixes:
Bug Fixes:
Fixed table facility when updates to Units of Measure occur while one slave has the updated table open. (30330)
Version 4.7.1.13
Bug Fixes:
Fixed xmim_move utility to support intraday data.
Fixed issue with intraday continuous contracts where contracts are in different
database stores.
Version 4.7.1.12
Added timeout option "-t" to xmim_get_rel_paths. (59399)
Bug Fixes:
Fixed publisher handling of doubles in contract relation columns.
Fixed issue with writer_wait command to only sync with databases that are not read-only.
Fixed a problem with corrections where it was failing to load initially at times.
Version 4.7.1.9
Fixed Linux issue where the string 'nan' was being printed when 'NaN' was expected.
Version 4.7.1.7
Now syncing schema when unlocking schema to maintain schema
integrity. (58483, 57904)
Added large block size support for Composite columns.
Bug Fixes:
Fixed publisher causing problems when publishing during a rollover.(58610)
Fixed publisher issue where new relation columns were added
and all subsequent data where shown as CREATE as opposed to the expected INSERT.
Fixed publisher issue where deleted rows showed bad values
in log.
Fixed publisher issue where UPDATES old values contained
bad values.
Version 4.7.1.6
Fixed issue with the entitlement gui. When an entitlement name (user/group id) is removed from a list, it would be removed from all lists (groups/entitlements). This is now resolved.
Removed DST optimization code introduced in 4.7.1 because it adversely affects “get records” operations.
Fixed xmim_server issue involving a “too long” hostname with the “-q” option.
Fixed publisher handling of NaN values. (56614)
Futures data can now be “entitled’. Previously, it was a stated/document rule that all futures was given a “free pass”. With the new entitlement performance enhancement, customers may entitle/manage futures data access.
Version 4.7.1.3
Fixed bmim_client parsing of command line. The command ‘bmim_client’
no longer assumes every invalid option entry is a file to parse. (XXXXXX)
Version 4.7.1.2
Fixed memory issues.
Version 4.7.1.1
Fixed builtin issue when it encountered a date/time that was
“invalid”. (54422)
Version 4.7.1
Optimized entitlement verification for user/group lookups. Very
large entitlement lookups could take minutes. With this fix, the same lookup
takes seconds. This fix requires reloading the users entitlement database if
upgrading from a 4.6.x or older version. (46257)
Added publisher to publish deltas for continuous daily contracts. Provided option for .xmimrc called “echo continuousLoggingP: yes” to turn on option. Default behavior is off. (48962)
Added builtin C stored function to stitch two series together. Fills missing data according to Execute Options if there is a “gap”. (XXXXXX)
Bug Fixes
Version 4.6.34.2
Added ‘%exec.within_range’ option, which allows users a query “over
the last 30 seconds”.
Version 4.6.34.1
Fixed real-time data with double columns. (49858)
Fixed reshaping code to handle midnight data point correctly. (52522)
Added a option for .xmimrc “echo PublishExactPrn: yes|no” to turn on logging directly the value found in the facts_read prn file. Default value is off (no). This option does not effect put_records operation. (46779)
Added API XmimGetRecordsOptionDouble() and XmimVaGetRecordsOptionDouble(). (46308)
Bug Fixes
Fixed xmim_move utility application for sub_day data.
Fixed publisher issue for x86 in put_records.
Removed block size limitation for composite relations.
Version 4.6.33.15
Fixed publisher with regen_continuous data (53815)
Version 4.6.33.8
Fixed publisher with regen_continuous data.
Version 4.6.33.7
Fixed unexpected behavior when trying to verify package.
Fixed millisecond query using “1 value ago”. (50089)
Fixed RPC errors. (51029)
Version 4.6.33.3
Fixed P/L query order can enter and exit in same unit. (48690)
Version 4.6.33.2
Fixed P/L query not exiting “1 day after entry_day”. (46476)
Version 4.6.33.1
Reverted optimizations on reshaping issue.
Version 4.6.33
Bug Fixes.
Version 4.6.31
Bug Fixes.
Added change to server http support to assume localhost for the LET http syntax. (42783)
Added “geometric backward adjusted prices” to rollover policy. (41425)
Bug Fixes.
Fixed an issue with 64bit Linux servers if their hosted begins with 0x8 or greater.
Added new API calls; XmimGetRelsForFile() and XmimGetSizeForRelcols(). (50216)
Fixed facts read failure. (50808)
Fixed bug introduced in 4.6.29.04. Added a comma between the old value and the new value for the corrected prices. (45050)
Added new field for publisher on Updates which will represent the old value.
Added option to .xmimrc “auto_regen: yes|no” (39838)
Bug Fixes.
Version 4.6.29.1
Implemented work-around for a solaris 10 bug.
Version 4.6.29
Added
http support to the LET statement in the query language. (42783)
Version 4.6.27.15
Optimized entitlements.
Bug Fixes.
Fixed Publisher. (47544)
Added .xmimrc option “PublishExactPrn: yes|no” to print the value as shown in the prn file read by the facts_read. Default is no. (46779)
Fixed XmimGetRecords() API
Removed change in rollover that affected volume and openinterest crossover queries.
Added syntax “Monday of expiration week” and “Monday before expiration week” to rollover date. (26952)
Added composite column support for mimdbcp().
Relation_add works with milliseconds. (29793)
Added corrections data within the publisher for all updates. (30311)
Added new API XmimGetRelDescriptions() to get descriptions for an array of relations or columns. (27380)
Enabled logging for XmimPutRecords(). (27595)
Added two new APIs; XmimGetRecordsDouble() and XmimPutRecordsDouble().
Bug Fixes.
Added batch_get_relation() utility application.
Added APIs XmimGetRelsForFile() and XmimGetRelsForFile(). (48239)
Added new API XmimGetDataRangeForTimePoint(). (30371)
Added new parameter XMIM_TIME_RANGE_TYPE for API XmimVaGetRecords(). (30372)
Bug Fixes.
Added BMIM command writer_wait. (29753)
Added user how many contracts to roll with “volume crossover” and “open interests crossover” policies. (25167)
Added xmim_pairs_gen() as an utility application. (29185)
Added suppression of “Relation does not exist” error message in bmim update scripts for relation_alias_add BMIM command. (29408)
Bug Fixes
Version 4.6.16
Fixed query when query is sent to a read-only database. (29561)
Version 4.6.15
Fixed API XmimVaPutRecords(). It did not process the milliseconds in the trading-range which is specified by XMIM_FROM_TIME and XMIM_TO_TIME.
Version 4.6.14
Added publisher logging for the put_records() API call. (28209)
Added new option ‘%exec.num_days_per_week’ to fix trading pattern problem. ‘%exec.num_days_per_week’ can be 5 or 7. Default value is 0. 0 means no behaviorior change. (29071)
Version 4.6.11
Provide user with the capability to skip contracts during rollover. Mechanism to skip contracts is for the user to specify identical expiration dates. (28253)
Fixed linux 32 bit failure when a full log file is encountered. (29071)
Version 4.6.10
Modified XmimFindRelations() and XmimVAFindRelations() to limit searches. (27852)
Version 4.6.8
Added inheritance for trading_pattern, start_time, and end_time, where these attributes can be inherited from the parent category when adding a relation. (27426)
Version 4.6.5
Added new option ‘%exec.uniontime”. (23695)
Version 4.6.1
Added %exec.strictvariablenames option. (21418)
Added “disablebackadjustment” feature to .xmimrc. (22900)
Added new API XmimGetNumRecords(). (22124)
Added restriction for variables not. Variables can not contain ‘$’ or ‘@’. (21418)
Added an option “-N” to xmim_get_options() API command to include a column for the the contract name per a customer request. (24072)
Modified API xmim_get_constituents() arguments so that old component lists can be created. (24210)
Removed ‘Other’ category from database initial creation. (21992)
Added BMIM database_def_write command. It creates a file with defs for the database currently narrowed to. Additional commands added are “print_daily_file”, “print_intraday_file”, and “print_tick_file”, which retrieve and print actual relcol location. (18087)
Added support for millisecond queries.
Bug Fixes.
Version 4.6
1. Forward Curve Functionality
2. Second and millisecond data accessible from the MIM query language
3. Composite columns accessible from the MIM query language
4. C Stored Functions
Future Enhancements:
64-bit support for Linux
Version 4.5.26
NEW QUERY LANGUAGE BUILTINS:
mid_point
median
dividend_adjusted_return
compound_return
NEW QUERY EXECUTION OPTIONS:
%exec.undef_variable_value, undefined variable initial value, default is 0.0
%exec.process_nan_values, value of NO can boost query performance because it doesn't process NaN values, default is YES.
NEW BMIM COMMANDS:
suppress_relation_not_found
check_relation_not_found
entitlements_write_all
MILLISECOND DATA SUPPORT:
Support for millisecond timeseries.
Added BMIM commands: database_def_millisecond_file.
XmimPutRecords() supports millisecond data.
XmimGetRecords() supports millisecond data.
xmim_get application, and new application, xmim_put, support millisecond data.
NOTE: - Queries with millisecond data are supported in 4.6.x.
64-BIT SUPPORT:
64-bit -vs- 32-bit information is provided in xmim_svr_info (server.info)
and in "bmim_client -v".
64-bit support is currently for Solaris only.
CORRECTIONS FACILITY:
Corrections facility now works properly.
PORTMAPPER CHANGES / EXPLICIT PORTS:
Stop use of portmapper for high ports.
Added -M flag to xmim_server and xmim_slave_server to stop
registration with the portmapper.
Port numbers are specified as usual (-p for master, -n for slave)
when starting the server.
Ports 0-128 are standard ports and are assigned to the portmapper.
Ports 1056-127968 are explicit ports. Typically values will be in the
range 1056 to 12000 every 32. The TCP port is mapped back to an
RPC program number, computed as follows: (tcp port - 1024)/32 + 128.
The ports are used in 32 port blocks.
Example: For static port 8128:
Start server with this command: xmim_server -p 8128 -M
TCP ports will be:
8128=master,
8129=4091 writer slave,
8130=tick writer,
8128+4=slave 0,
8128+5=slave 1, ...
RPC program number will be (8128 - 1024)/32 + 128 = 350
CUSTOMER ENHANCEMENTS:
NEW QUERY LANGUAGE BUILTINS:
. Incident ID: 13870
Adding mid_point as builtin function for the query language. It is exactly the same as midpoint.
This fixes a problem when there is also a "midpoint" column that conflicts with the name of the "midpoint" builtin.
. Incident ID: 13350
Adding new builtin function, median, to the query language.
. Incident: 19595
Added a new built-in function, dividend_adjusted_return, to the query language.
. Incident: 19080
Added a new built-in function, compound_return, to the query language.
NEW QUERY EXECUTION OPTIONS:
. Incident ID: 16489
Adding execution option %exec.undef_variable_value. Default value is 0.0.
Set this variable if you wish to provide a different default value for undefined
variables. (example, %exec.undef_variable_value: NaN)
. Incident ID: 16439
Adding execution option, %exec.process_nan_values. Default value is YES.
%exec.process_nan_values: NO can greatly improve query performance by skipping over all NaN values
(i.e. not processing them). For some queries,skipping NaN values is OK. But, for other queries, this
could yield incomplete results.
OTHER:
. Incident ID: 14078
Setting the bmim relation_add command trade pattern default
value.
. Incident: 22394
Change master/slave behavior to attempt to kill heavily-utilized slaves for
a "fresh start".
. Incident: 21891
Customers need a way to list the unaliased constituents of a category name.
Therefore, we added a utility program, xmim_get_constituents.
. Incident ID: 16761
Adding relation name to time series error message.
CUSTOMER BUG FIXES:
. Incident ID: 16037
Fix the Macro editor save same macro multi-time bug.
. Incident ID: 15915
Fix "Can not read time series from disk" problems.
. Incident ID: 17580
facts_write could not read composite column data.
. Incident ID: 17823
Changed code for adding relation so that relation-names with
colon character can not be added.
. Incident: 18192
Martin Luther King Day was not recognized as a holiday.
. Incident: 17783
xmim_client could not connect to server after it was restarted.
. Incident: 18721
Could not get intraday data when end time is after trade end time.
. Incident: 18721
Could not get intraday data when end time is after trade end-time.
This is refix for other relations.
OTHER FIXES:
. Fixed api calls for XmimGetSvrCustomSearch, XmimGetSvrTransformers,
XmimGetSvrMacros. Calls previously were looping, but now can stop
looping by checking the result vector length.
. Fixed BMIM formula_add command.
. Removed a flag that was causing some of the continuous contracts
to not be regenerated by regen_continuous.
. Fixed display of error message for lock_files.
. Server assumed . and .. would be first directories from readdir. On
Linux 2.6 kernel this is not true. Server had inifinite loop on startup.
. Remove temporary files when doing corrections.
. xmim_client could not graph data file.
. XmimLoadBmimScript did not work when given a query to execute,
due to lack of initialization.
. Fix bug in entitlements where the "root" user (the server owner) was not getting a
free pass during certain BMIM operations such as relation_modify.
. Added changes to attempt to avoid 4091 race conditions.
. Extended the size of error messages from 512 to 1024 bytes.
. Fixed problem with Linux entitlements_write command in BMIM.
OTHER ENHANCEMENTS:
. Added column name to the output of the new xmim_get_impvol utility.
. BMIM now has seperate error messages for Invalid Relation, Duplicate Relation,
Invalid Column, and Duplicate Column. Previously, the Relation messages
were combined into one message so that you did not know if the relation was
invalid or duplicate. The previous column message had the same problem.
. Added database name to some of the "Relation does not exist"
error messages.
. Added 2 new BMIM commands:
#v4.5 suppress_relation_not_found
#v4.5 check_relation_not_found
These new commands suppress/unsuppress the "Relation does not exist" message
from the relation_modify command. (only for the relation_modify command)
. Change xmim_find to work with leave relations and partial paths.
Will now read relations from stdin. So, a file of symbols can be used.
echo "CL_2004Z" | xmim_find -o np -
. Use svc_run for program loops to allow large number of FD's.
. Changes to xmim_find:
a) Changed xmim_find to use full pathnames of relations so that
duplicate relations do not make it stop processing.
b) Changed the slave to remove a UMR and also make a better error
message when duplicate relations are found.
c) Changed xmim_find to disconnect and exit when there is an
error.
. Enhanced the 'lock_files' failure message to give information
about the server owner. Also, give a reason code for failure.
. Added improved server_wrapper::loop() logic back in
so that we can call kill_expired_slaves() in the loop.
Also, got rid of the 1024 file limit.
. Better error message for the 'SchSchemaSingle::setupRelcol failed'
message.
. Removed the Sort system call which fixed Corrections
. Removed need for oncrpc.lib and oncrpc.dll. oncrpc support
is now built into xmim4.dll and xmim4.lib. This reduces confusion
with DLL's at customer sites.
. Added new argument to the XmimVaFindRelations() function "XMIM_INCLUDE_HIDDEN".
This allows a programmer to find ".Internal" relations when set to true.
By default it is false, the ".Interal" relations are excluded
from the results.
. Added bmim command entitlements_write_all.
. When translating secondly data to minutely data,
facts_read could not read the intraday data (with TICK_CONVERT)
with timestamp at a boundary containing seconds.
Example: 23:59:05 was translated to 24:00 (not understood).
Now it is translated to 00:00.
MIMDBCP:
. Added switch to MIMDBCP utility to only copy the futures contracts
and not the continuous contracts. After copying the futures contracts,
the regen_continuous function is called to generate the continuous
contracts.
. mimdbcp need to default to original behavior without requiring .cfg file.
Set default for contracts-only? to false.
. Fixed a bug that caused mimdbcp to crash when both verbose and
start date were not set.
Changed the algorithm to not copy continuous data, instead
XmimRegenRollover is called on each relation that has continuous children.
. Added the from_date example to the config file.
DATABASE SCHEMA STABILITY:
. Gracefully handle the error where you have a partial-write that was interrupted,
but now can't be completed due to an error. Abandon the completion of the
partial-write and continue on to the newly-requested write processes.
(Previously, the partial-write continued in places where it shouldn't,
potentially damaging the database schema file).
(also fixed in 4.4.23)
. Closed up a hole that was allowing hash-tables to write out past their curSize.
Fixed infinite loop. Also, made hash-tables generally bigger for new databases,
especially the relcol_property tables.
. Fixing problem found where loading an mjk package causes
"Can not read time series from disk" error. Here are the fixes:
a) Relation name was incorrect or often missing in the
'Unable to Read the Time Series %s from disk' error message.
Also, added the column name to the message.
b) Fixed bad place when reading timeseries where offset got off by 4 bytes.