Advanced Topic: Using External C++ Functions#
This is based on the relevant portion of the CmdStan documentation here
Consider the following Stan model, based on the bernoulli example.
[2]:
from cmdstanpy import CmdStanModel
model_external = CmdStanModel(stan_file='bernoulli_external.stan', compile=False)
print(model_external.code())
As you can see, it features a function declaration for make_odds
, but no definition. If we try to compile this, we will get an error.
[3]:
model_external.compile()
18:46:02 - cmdstanpy - INFO - compiling stan file /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external.stan to exe file /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external
18:46:18 - cmdstanpy - INFO - compiled model executable: /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external
18:46:18 - cmdstanpy - WARNING - Stan compiler has produced 1 warnings:
18:46:18 - cmdstanpy - WARNING -
--- Translating Stan model to C++ code ---
bin/stanc --o=/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external.hpp /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external.stan
Warning: Empty file
'/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external.stan'
detected; this is a valid stan model but likely unintended!
--- Compiling, linking C++ code ---
/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.2/bin/x86_64-conda-linux-gnu-c++ -std=c++1y -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes -I /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.2/include/ -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_6.0.0/include -I stan/lib/stan_math/lib/sundials_6.0.0/src/sundials -DBOOST_DISABLE_ASSERTS -DTBB_INTERFACE_NEW -DTBB_INTERFACE_NEW -c -x c++ -o /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external.o /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external.hpp
/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.2/bin/x86_64-conda-linux-gnu-c++ -std=c++1y -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes -I /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.2/include/ -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_6.0.0/include -I stan/lib/stan_math/lib/sundials_6.0.0/src/sundials -DBOOST_DISABLE_ASSERTS -DTBB_INTERFACE_NEW -DTBB_INTERFACE_NEW -Wl,-L,"/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.2/lib/" -Wl,-rpath,"/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.2/lib/" -Wl,--disable-new-dtags -ltbb /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external.o src/cmdstan/main.o -Wl,-L,"/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.2/lib/" -Wl,-rpath,"/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.2/lib/" -Wl,--disable-new-dtags -ltbb stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_nvecserial.a stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_cvodes.a stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_idas.a stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_kinsol.a -o /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external
rm -f /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external.o
Even enabling the --allow-undefined
flag to stanc3 will not allow this model to be compiled quite yet.
[4]:
model_external.compile(stanc_options={'allow-undefined':True})
18:46:18 - cmdstanpy - INFO - compiling stan file /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external.stan to exe file /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external
18:46:19 - cmdstanpy - WARNING - Stan compiler has produced 1 warnings:
18:46:19 - cmdstanpy - WARNING -
--- Translating Stan model to C++ code ---
bin/stanc --allow-undefined --o=/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external.hpp /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external.stan
Warning: Empty file
'/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external.stan'
detected; this is a valid stan model but likely unintended!
--- Compiling, linking C++ code ---
/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.2/bin/x86_64-conda-linux-gnu-c++ -std=c++1y -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes -I /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.2/include/ -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_6.0.0/include -I stan/lib/stan_math/lib/sundials_6.0.0/src/sundials -DBOOST_DISABLE_ASSERTS -DTBB_INTERFACE_NEW -DTBB_INTERFACE_NEW -c -include /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/user_header.hpp -x c++ -o /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external.o /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external.hpp
cc1plus: fatal error: /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/user_header.hpp: No such file or directory
compilation terminated.
make: *** [make/program:58: /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external] Error 1
Command ['make', 'STANCFLAGS+=--allow-undefined', '/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external']
error during processing No such file or directory
To resolve this, we need to both tell the Stan compiler an undefined function is okay and let C++ know what it should be.
We can provide a definition in a C++ header file by using the user_header
argument to either the CmdStanModel constructor or the compile
method.
This will enables the allow-undefined
flag automatically.
[5]:
model_external.compile(user_header='make_odds.hpp')
18:46:19 - cmdstanpy - INFO - compiling stan file /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external.stan to exe file /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external
18:46:34 - cmdstanpy - INFO - compiled model executable: /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external
18:46:34 - cmdstanpy - WARNING - Stan compiler has produced 1 warnings:
18:46:34 - cmdstanpy - WARNING -
--- Translating Stan model to C++ code ---
bin/stanc --allow-undefined --o=/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external.hpp /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external.stan
Warning: Empty file
'/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external.stan'
detected; this is a valid stan model but likely unintended!
--- Compiling, linking C++ code ---
/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.2/bin/x86_64-conda-linux-gnu-c++ -std=c++1y -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes -I /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.2/include/ -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_6.0.0/include -I stan/lib/stan_math/lib/sundials_6.0.0/src/sundials -DBOOST_DISABLE_ASSERTS -DTBB_INTERFACE_NEW -DTBB_INTERFACE_NEW -c -include /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/make_odds.hpp -x c++ -o /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external.o /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external.hpp
/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.2/bin/x86_64-conda-linux-gnu-c++ -std=c++1y -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes -I /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.2/include/ -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_6.0.0/include -I stan/lib/stan_math/lib/sundials_6.0.0/src/sundials -DBOOST_DISABLE_ASSERTS -DTBB_INTERFACE_NEW -DTBB_INTERFACE_NEW -Wl,-L,"/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.2/lib/" -Wl,-rpath,"/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.2/lib/" -Wl,--disable-new-dtags -ltbb /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external.o src/cmdstan/main.o -Wl,-L,"/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.2/lib/" -Wl,-rpath,"/home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/conda/v1.0.2/lib/" -Wl,--disable-new-dtags -ltbb stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_nvecserial.a stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_cvodes.a stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_idas.a stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_kinsol.a -o /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external
rm -f /home/docs/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/docsrc/examples/bernoulli_external.o
We can then run this model and inspect the output
[6]:
fit = model_external.sample(data={'N':10, 'y':[0,1,0,0,0,0,0,0,0,1]})
fit.stan_variable('odds')
18:46:34 - cmdstanpy - INFO - CmdStan start processing
18:46:34 - cmdstanpy - INFO - CmdStan done processing.
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Input In [6], in <cell line: 2>()
1 fit = model_external.sample(data={'N':10, 'y':[0,1,0,0,0,0,0,0,0,1]})
----> 2 fit.stan_variable('odds')
File ~/checkouts/readthedocs.org/user_builds/cmdstanpy/checkouts/v1.0.2/cmdstanpy/stanfit/mcmc.py:726, in CmdStanMCMC.stan_variable(self, var, inc_warmup)
684 """
685 Return a numpy.ndarray which contains the set of draws
686 for the named Stan program variable. Flattens the chains,
(...)
723 CmdStanGQ.stan_variable
724 """
725 if var not in self._metadata.stan_vars_dims:
--> 726 raise ValueError(
727 f'Unknown variable name: {var}\n'
728 'Available variables are '
729 + ", ".join(self._metadata.stan_vars_dims)
730 )
731 if self._draws.shape == (0,):
732 self._assemble_draws()
ValueError: Unknown variable name: odds
Available variables are
The contents of this header file are a bit complicated unless you are familiar with the C++ internals of Stan, so they are presented without comment:
#include <boost/math/tools/promotion.hpp>
#include <ostream>
namespace bernoulli_model_namespace {
template <typename T0__> inline typename
boost::math::tools::promote_args<T0__>::type
make_odds(const T0__& theta, std::ostream* pstream__) {
return theta / (1 - theta);
}
}