Advanced Topic: Using External C++ Functions#
This is based on the relevant portion of the CmdStan documentation here
Consider the following Stan model, based on the bernoulli example.
[2]:
from cmdstanpy import CmdStanModel
model_external = CmdStanModel(stan_file='bernoulli_external.stan', compile=False)
print(model_external.code())
As you can see, it features a function declaration for make_odds, but no definition. If we try to compile this, we will get an error.
[3]:
model_external.compile()
20:26:32 - cmdstanpy - INFO - compiling stan file /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external.stan to exe file /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external
20:26:47 - cmdstanpy - INFO - compiled model executable: /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external
20:26:47 - cmdstanpy - WARNING - Stan compiler has produced 1 warnings:
20:26:47 - cmdstanpy - WARNING -
--- Translating Stan model to C++ code ---
bin/stanc --o=/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external.hpp /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external.stan
Warning: Empty file
'/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external.stan'
detected; this is a valid stan model but likely unintended!
--- Compiling, linking C++ code ---
/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.3/bin/x86_64-conda-linux-gnu-c++ -std=c++1y -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes -I /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.3/include/ -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_6.0.0/include -I stan/lib/stan_math/lib/sundials_6.0.0/src/sundials -DBOOST_DISABLE_ASSERTS -DTBB_INTERFACE_NEW -DTBB_INTERFACE_NEW -c -x c++ -o /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external.o /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external.hpp
/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.3/bin/x86_64-conda-linux-gnu-c++ -std=c++1y -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes -I /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.3/include/ -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_6.0.0/include -I stan/lib/stan_math/lib/sundials_6.0.0/src/sundials -DBOOST_DISABLE_ASSERTS -DTBB_INTERFACE_NEW -DTBB_INTERFACE_NEW -Wl,-L,"/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.3/lib/" -Wl,-rpath,"/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.3/lib/" -Wl,--disable-new-dtags -ltbb /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external.o src/cmdstan/main.o -Wl,-L,"/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.3/lib/" -Wl,-rpath,"/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.3/lib/" -Wl,--disable-new-dtags -ltbb stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_nvecserial.a stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_cvodes.a stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_idas.a stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_kinsol.a -o /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external
rm -f /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external.o
Even enabling the --allow-undefined flag to stanc3 will not allow this model to be compiled quite yet.
[4]:
model_external.compile(stanc_options={'allow-undefined':True})
20:26:47 - cmdstanpy - INFO - compiling stan file /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external.stan to exe file /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external
20:26:47 - cmdstanpy - WARNING - Stan compiler has produced 1 warnings:
20:26:47 - cmdstanpy - WARNING -
--- Translating Stan model to C++ code ---
bin/stanc --allow-undefined --o=/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external.hpp /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external.stan
Warning: Empty file
'/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external.stan'
detected; this is a valid stan model but likely unintended!
--- Compiling, linking C++ code ---
/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.3/bin/x86_64-conda-linux-gnu-c++ -std=c++1y -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes -I /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.3/include/ -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_6.0.0/include -I stan/lib/stan_math/lib/sundials_6.0.0/src/sundials -DBOOST_DISABLE_ASSERTS -DTBB_INTERFACE_NEW -DTBB_INTERFACE_NEW -c -include /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/user_header.hpp -x c++ -o /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external.o /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external.hpp
cc1plus: fatal error: /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/user_header.hpp: No such file or directory
compilation terminated.
make: *** [make/program:58: /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external] Error 1
Command ['make', 'STANCFLAGS+=--allow-undefined', '/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external']
error during processing No such file or directory
To resolve this, we need to both tell the Stan compiler an undefined function is okay and let C++ know what it should be.
We can provide a definition in a C++ header file by using the user_header argument to either the CmdStanModel constructor or the compile method.
This will enables the allow-undefined flag automatically.
[5]:
model_external.compile(user_header='make_odds.hpp')
20:26:47 - cmdstanpy - INFO - compiling stan file /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external.stan to exe file /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external
20:27:03 - cmdstanpy - INFO - compiled model executable: /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external
20:27:03 - cmdstanpy - WARNING - Stan compiler has produced 1 warnings:
20:27:03 - cmdstanpy - WARNING -
--- Translating Stan model to C++ code ---
bin/stanc --allow-undefined --o=/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external.hpp /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external.stan
Warning: Empty file
'/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external.stan'
detected; this is a valid stan model but likely unintended!
--- Compiling, linking C++ code ---
/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.3/bin/x86_64-conda-linux-gnu-c++ -std=c++1y -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes -I /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.3/include/ -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_6.0.0/include -I stan/lib/stan_math/lib/sundials_6.0.0/src/sundials -DBOOST_DISABLE_ASSERTS -DTBB_INTERFACE_NEW -DTBB_INTERFACE_NEW -c -include /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/make_odds.hpp -x c++ -o /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external.o /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external.hpp
/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.3/bin/x86_64-conda-linux-gnu-c++ -std=c++1y -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes -I /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.3/include/ -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_6.0.0/include -I stan/lib/stan_math/lib/sundials_6.0.0/src/sundials -DBOOST_DISABLE_ASSERTS -DTBB_INTERFACE_NEW -DTBB_INTERFACE_NEW -Wl,-L,"/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.3/lib/" -Wl,-rpath,"/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.3/lib/" -Wl,--disable-new-dtags -ltbb /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external.o src/cmdstan/main.o -Wl,-L,"/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.3/lib/" -Wl,-rpath,"/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.3/lib/" -Wl,--disable-new-dtags -ltbb stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_nvecserial.a stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_cvodes.a stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_idas.a stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_kinsol.a -o /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external
rm -f /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/docsrc/examples/bernoulli_external.o
We can then run this model and inspect the output
[6]:
fit = model_external.sample(data={'N':10, 'y':[0,1,0,0,0,0,0,0,0,1]})
fit.stan_variable('odds')
20:27:03 - cmdstanpy - INFO - CmdStan start processing
20:27:03 - cmdstanpy - INFO - CmdStan done processing.
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Input In [6], in <cell line: 2>()
1 fit = model_external.sample(data={'N':10, 'y':[0,1,0,0,0,0,0,0,0,1]})
----> 2 fit.stan_variable('odds')
File ~/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.3/cmdstanpy/stanfit/mcmc.py:729, in CmdStanMCMC.stan_variable(self, var, inc_warmup)
687 """
688 Return a numpy.ndarray which contains the set of draws
689 for the named Stan program variable. Flattens the chains,
(...)
726 CmdStanGQ.stan_variable
727 """
728 if var not in self._metadata.stan_vars_dims:
--> 729 raise ValueError(
730 f'Unknown variable name: {var}\n'
731 'Available variables are '
732 + ", ".join(self._metadata.stan_vars_dims)
733 )
734 if self._draws.shape == (0,):
735 self._assemble_draws()
ValueError: Unknown variable name: odds
Available variables are
The contents of this header file are a bit complicated unless you are familiar with the C++ internals of Stan, so they are presented without comment:
#include <boost/math/tools/promotion.hpp>
#include <ostream>
namespace bernoulli_model_namespace {
template <typename T0__> inline typename
boost::math::tools::promote_args<T0__>::type
make_odds(const T0__& theta, std::ostream* pstream__) {
return theta / (1 - theta);
}
}