Sapling SAP
Reference

The R twin of the SAS macro library

One R function for each SAS macro: the same blocks, in the same order, with the same labels.

This page is the repository's own document docs/r-twin.md, shown as it is, so that it never drifts from the product.

A statistician or a programmer who knows the SAS macros of sas/macros/ must be able to read the R program of the same table, and the other way round. The R library r/sapling/ has one R function for each SAS macro. A program of the R library reads like the SAS program of the same table: the same blocks, in the same order, with the same labels.

Three slices are made and proven:

  1. describe, freq, tlf_stack, columns, ard, tlf_layout, tlf_print, tlf_table and table_summary (and study with adam(), the two small parts every program needs).

  2. The tests and the models, and the pairs: ttest, wilcoxon, chisq, fisher, cmh, tlf_anova, ancova, lsmeans, adjust.

  3. worst, termflags, proportion, risk, rate, rateratio, and the one-call tables table_ae, table_visit, table_shift, table_response. They share one program writer (r/sapling/_spl_program.R), as the SAS macros share %_spl_program.

  4. The time to the event and the models: km, logrank, cox, table_tte, logistic (and table_response(covars=)), mmrm, table_model, the four interaction_*; and the listings of records: csv and tlf_print of records.

  5. The RTF of tlf_print() and the nine figures (base graphics).

  6. The ADaM steps and import_xpt.

All 94 macros are accounted for in the table below: done, or not needed with the reason (the inner parts that SAS needs and R does not).

All data of the proof are the made-up data of the study SYNHTN01. No number here is a result of a study.

The idea#

The naming rule#

The R function has the name of the macro. Where base R (base, stats, utils, graphics, methods) already has a function of that name, the R name is tlf_<name>. A test (test/r_twin.test.ts) checks every macro name with exists() in a vanilla R session. These seven are the exceptions:

MacroR functionBase R already has
%stacktlf_stackutils::stack
%layouttlf_layoutgraphics::layout
%printtlf_printbase::print
%tabletlf_tablebase::table
%anovatlf_anovastats::anova
%barplottlf_barplotgraphics::barplot
%boxplottlf_boxplotgraphics::boxplot

The inner helpers of the R library start with .spl_ and sit in the files that start with _spl_, as the inner macros do.

How an argument is given#

One table program, side by side#

Table 14.2.1 of SYNHTN01 (the ITT population). On the left the SAS program out/sas/programs/t14_2_1.sas, on the right r/sapling/examples/t14_2_1.R.

SAS:

%describe(data=adsl, var=age, by=trt01p, total=yes, label="Age (years)", out=t14_2_1_age_summ)

%freq(
    data   = adsl
  , var    = agegr1
  , by     = trt01p
  , keep   = agegr1n
  , levels = agegr1 '<65' '65-<75' '>=75'
  , total  = yes
  , label  = "Age group (years)"
  , out    = t14_2_1_agegr_summ
)

%freq(data=adsl, var=sex, by=trt01p, total=yes, label="Sex", out=t14_2_1_sex_summ)

%stack(t14_2_1_age_summ t14_2_1_agegr_summ t14_2_1_sex_summ, out = t14_2_1)

%columns(data=t14_2_1, n=adsl, var=trt01p, order=trt01pn, total=yes, out=t14_2_1_cols)
%ard(data=t14_2_1, columns=t14_2_1_cols)
%layout(
    data    = t14_2_1
  , columns = t14_2_1_cols
  , var     = trt01p
  , format  = sex $sexf.
  , file    = &spl_out./tlf/t14_2_1
  , setup   = t14_2_1_print
  , out     = t14_2_1_display
)
TITLE1 "Table 14.2.1 Demographic and Baseline Characteristics";
FOOTNOTE1 J=l "Program: t14_2_1.sas   Output: T14_2_1";
%print(&t14_2_1_print.)

R:

t14_2_1_age_summ <- describe(data = adsl, var = age, by = trt01p, total = "yes", label = "Age (years)")

t14_2_1_agegr_summ <- freq(
    data   = adsl
  , var    = agegr1
  , by     = trt01p
  , keep   = agegr1n
  , levels = list(agegr1 = c("<65", "65-<75", ">=75"))
  , total  = "yes"
  , label  = "Age group (years)"
)

t14_2_1_sex_summ <- freq(data = adsl, var = sex, by = trt01p, total = "yes", label = "Sex")

t14_2_1 <- tlf_stack(t14_2_1_age_summ, t14_2_1_agegr_summ, t14_2_1_sex_summ)

t14_2_1_cols <- columns(data = t14_2_1, n = adsl, var = trt01p, order = trt01pn, total = "yes")
ard(data = t14_2_1, columns = t14_2_1_cols)
t14_2_1_display <- tlf_layout(
    data    = t14_2_1
  , columns = t14_2_1_cols
  , var     = trt01p
  , format  = list(sex = c(F = "Female", M = "Male", U = "Unknown"))
  , file    = file.path(spl_out, "tlf", "t14_2_1")
  , setup   = "t14_2_1_print"
)
tlf_print(
    t14_2_1_print
  , title    = "Table 14.2.1 Demographic and Baseline Characteristics"
  , footnote = "Program: t14_2_1.sas   Output: T14_2_1"
)

The same table in one call (R; the SAS call is %table_summary(...) with the same parameters):

table_summary(
    data   = adsl
  , where  = ittfl == "Y"
  , vars   = "age agegr1#agegr1n sex# race# ethnic#"
  , by     = trt01p
  , order  = trt01pn
  , total  = "yes"
  , levels = list(agegr1 = c("<65", "65-<75", ">=75"))
  , file   = file.path(spl_out, "tlf", "t14_2_1")
)

Like the macro, table_summary() writes the long R program (one call of describe() or freq() for each variable, tlf_stack(), tlf_table()) and runs it. show = "yes" prints the program; show = "<file>" writes it to a file.

The correspondence: the 94 macros#

"1: done" to "6: done" are made and proven (the slice). "Planned" is not made yet. The inner _spl_ macros of SAS are listed with their R helper, or as not needed.

Setting up a program#

SAS macroR functionSlice
%import_xptimport_xpt6: done
%studystudy1: done

The statistics macros, the tables and the steps of a table#

SAS macroR functionSlice
%adjustadjust2: done
%ancovaancova2: done
%anovatlf_anova2: done
%ardard1: done
%chisqchisq2: done
%cmhcmh2: done
%columnscolumns1: done
%coxcox4: done
%csvcsv4: done
%describedescribe1: done
%fisherfisher2: done
%freqfreq1: done
%interaction_genmodinteraction_genmod4: done
%interaction_glminteraction_glm4: done
%interaction_logisticinteraction_logistic4: done
%interaction_phreginteraction_phreg4: done
%kmkm4: done
%layouttlf_layout1: done
%logisticlogistic4: done
%logranklogrank4: done
%lsmeanslsmeans2: done
%mmrmmmrm4: done
%proportionproportion3: done
%raterate3: done
%rateratiorateratio3: done
%riskrisk3: done
%tabletlf_table1: done
%table_aetable_ae3: done
%table_modeltable_model4: done
%table_responsetable_response3: done
%table_shifttable_shift3: done
%table_summarytable_summary1: done
%table_ttetable_tte4: done
%table_visittable_visit3: done
%termflagstermflags3: done
%ttestttest2: done
%wilcoxonwilcoxon2: done
%worstworst3: done

Printing#

SAS macroR functionSlice
%printtlf_print1: done
%stacktlf_stack1: done

Figures#

SAS macroR functionSlice
%aeplotaeplot5: done
%barplottlf_barplot5: done
%boxplottlf_boxplot5: done
%forestplotforestplot5: done
%kmplotkmplot5: done
%lineplotlineplot5: done
%meanplotmeanplot5: done
%profileplotprofileplot5: done
%scatterplotscatterplot5: done

The steps of an ADaM program#

SAS macroR functionSlice
%baselinebaseline6: done
%flag_firstflag_first6: done
%flag_if_anyflag_if_any6: done
%locflocf6: done
%merge_subjectmerge_subject6: done
%param_wideparam_wide6: done
%pickpick6: done
%poolpool6: done
%relchangerelchange6: done
%summarizesummarize6: done
%summarize_groupssummarize_groups6: done
%ttette6: done
%visitsvisits6: done
%windowswindows6: done

The inner parts (a person does not call them)#

SAS macroR functionSlice
%_spl_argnot needed: the program writer builds the calls itself (.spl_prog_call)not needed
%_spl_beginnot needed: no begin line (R messages carry the NOTE lines)not needed
%_spl_calldone: .spl_prog_calldone
%_spl_charsnot needed: is.character()not needed
%_spl_csvdone: .spl_csvdone
%_spl_emptydone: .spl_emptydone
%_spl_endnot needed: no end linenot needed
%_spl_figure_attrsdone: .spl_fig_* (_spl_figure.R)done
%_spl_figure_closedone: .spl_fig_* (_spl_figure.R)done
%_spl_figure_dodgedone: .spl_fig_* (_spl_figure.R)done
%_spl_figure_labeldone: .spl_fig_labeldone
%_spl_figure_nodatadone: .spl_fig_nodatadone
%_spl_figure_opendone: .spl_fig_* (_spl_figure.R)done
%_spl_figure_outpardone: .spl_fig_outpardone
%_spl_figure_pardone: .spl_fig_par (draw again from par=)done
%_spl_levelsdone: .spl_levelsdone
%_spl_needdone: .spl_needdone
%_spl_nobsnot needed: nrow()not needed
%_spl_oneofdone: .spl_oneofdone
%_spl_pairlistdone: .spl_pairlistdone
%_spl_pairsdone: .spl_pairsdone
%_spl_programdone: .spl_prog_*done
%_spl_quotenot needed: an R string is its own quotenot needed
%_spl_rowsdone: .spl_rowsdone
%_spl_stackpairsdone: .spl_stackpairsdone
%_spl_textdone: .spl_textdone
%_spl_totaldone: .spl_totaldone
%_spl_varsdone: .spl_varsdone
%_spl_windowsdone: .spl_windowsdone

What the library does the way SAS does#

The proof#

node tools/r_twin.ts has these parts. No SAS is started.

1. Twelve tables and three listings of SYNHTN01 against the recorded SAS run#

The R program of each table (r/sapling/examples/<table>.R, line by line against out/sas/programs/<table>.sas) is run with Rscript --vanilla and compared with the recorded SAS run (.scratch/sas_finish/synhtn01). The R programs read the ADaM data of that run (out/adam/data/*.rds).

TableFunctions it usesRows of the numbers fileListing lines
T14.1.1 Analysis populationsfreq with levels1514
T14.1.2 Subject dispositionfreq3317
T14.2.1 Demographicsdescribe, freq (levels, keep, total)8132
T14.3.1 SBP changedescribe, ancova (the p value in the column "All groups")3927
T14.4.1 Exposuredescribe3826
T14.5.1 Overview of TEAEsfreq with denom= and where2214
T14.5.2 TEAEs by SOC and PTfreq with two levels and events6228
T14.5.3 Serious TEAEsthe same2618
T14.7.1 Vital signs by visitdescribe with three groups and keep = c(param, paramn, avisitn), format of paramcd254153
T90.1 Worst severity, risk difference, rateworst, freq (threshold, order="frequency", orderof, levels), termflags, risk (newcombe), rate; three pages146103
T90.2 Time to the first TEAEdescribe, cox (efron, both directions of a pair)2222
SG90_T Cox model by sexfreq, cox with within, interaction_phreg, the p value in "All groups"2128

The listings of records L14.5.2, L14.5.3 and L90.1 (csv() and tlf_print() of records) are compared with their .lst and their .csv line by line: 189, 35 and 189 lines. They have the labels of the variables as headers, dates as DATE9., the columns that do not fit in 160 characters in a second part with the BY columns repeated, and one blank line and the titles between the parts.

A table passes when:

Together: 12 tables, 759 numbers, 910 listing lines, 3 listings of records.

Each of the 15 outputs also has its RTF file (.rtf, written by tlf_print() next to the .lst, see "The RTF" below). The text of its cells in order (titles, header cells, body cells, footnotes) is the same as that of the .rtf of SAS: 15 of 15 files.

2. The one-call tables against their long programs#

r/sapling/examples/t14_2_1_summary.R and r/sapling/examples/tables_oneshot.R. Each of table_summary, table_ae, table_visit, table_shift, table_response (with and without covars), table_tte and table_model (ANCOVA and MMRM) writes its long program (show = "<file>" keeps it); the program is run on its own; the display, the stacked blocks and the columns are identical data frames. table_summary is also identical to the display of t14_2_1.R and table_ae to that of t14_5_2.R (both proven against SAS). The population and the categories of the shift table and the responder flag of the response table are made up in that program (ADSL has no response; ADVS has no reference range).

3. The methods: the known answers#

r/sapling/examples/methods.R runs the R functions on the made-up fixture study of the method registry (FIXTURE_STUDY of src/methods.ts, 12 subjects), with the calls of each method's SAS fixture written in R. tools/r_twin.ts compares every number with the known answer of the registry, the answer the SAS macros were proven on (tools/methods_sas.ts; tolerance 1e-6, 1e-4 for the iterative methods): all 31 methods of the registry, 425 numbers.

Method (registry id)R function
count, contsumm, catsumm (with worst)freq, describe, freq + worst
anova, lsmeans, ancovatlf_anova, lsmeans, ancova
mmrm (Kenward-Roger and between-within)mmrm
chisq, fisher (two ways), cmh (two ways)chisq, fisher, cmh
ttest (three ways of giving the pairs), wilcoxonttest, wilcoxon
km, kmrate (log-log and linear), logrank (stratified or not)km, logrank
cox (efron, breslow, two strata variables, log-rank p value), logistic (covariate, class factor)cox, logistic
propciproportion (Clopper-Pearson)
riskdiff (newcombe; the four adjust methods and the weighted Bonferroni; the Mantel-Haenszel difference in strata), riskratio, oddsratio, pertermrisk, adjust, termflags
poisson, negbin, eair, eventraterateratio, rate
interaction-lm, -logistic, -poisson, -negbin, -coxinteraction_glm, interaction_logistic, interaction_genmod (both families), interaction_phreg

npm test has test/r_twin.test.ts: the name exceptions, the header box of each file, the argument names of each function against sas/macros/index.json, a few made-up subjects (the text of the numbers, the display rows, the messages of errors, the pairs, a Kaplan-Meier median that is NE, the refusal of a covariance structure or a df method the R library cannot make), the comparing tool itself, and the known answers (part 3).

4. The figures#

r/sapling/examples/figures.R runs the nine figure functions on the made-up data of tools/figures_sas.ts (the cases of FIGURES SAS), with the calls of the SAS cases in R. tools/r_twin.ts compares the numbers each function returns (the out= of the macro) with the out= of the recorded SAS run (.scratch/figures_sas/sas/out/<case>.csv) on the columns of that file, and with the known answers written by hand in tools/figures_sas.ts: 22 cases, 144 rows (km, forest, mean, median, line, line_band, box, bar, bar_stack, ae, profile, scatter, scatter_loess, and the empty cases, which have no row and the figure "No data qualified"). The PNG of each case is made (base graphics, png(), 300 dpi, 6.5in by 4.5in).

I looked at the pictures of kmplot, forestplot, meanplot, boxplot (beside the SAS picture, .scratch/figures_sas/sas/out/*.png), aeplot and barplot (the SAS picture of those two was not put beside), and made them drawn without error for lineplot, profileplot and scatterplot, which I did not look at after the last change of layout. I changed what was clearly worse: the median of kmplot is text in the panel as in SAS (not lines), the legend of kmplot has the entry "Censored", the names of the groups stand before the rows of the number at risk, the panels have the light grid and a grey box, the panels of a figure with panel= are stacked with a common axis label (the first drawing wasted space and clipped the label), the number table of meanplot is inside the panel, the forest plot has the end caps, the numbers at the right with the header "HR (CI)" and a legend of markers. What still differs in looks: the light grid of tlf_barplot is drawn over the bars; SAS draws a title (a TITLE statement, which the R functions do not take), a strip over each panel, filled markers of another size and its own fonts; the log axis of the forest plot has the ticks R chooses (0.5, 1, 2, 5), SAS 0.2, 0.5, 1, 2, 5; the colours, the markers and the line patterns of the groups are the same, in the order of columns=.

5. The ADaM steps#

r/sapling/examples/adam_steps.R runs the twin of each step macro on the fixtures of the step registry (src/adam_steps.ts, the fixtures that tools/adam_steps_sas.ts runs on SAS): the DATA steps around a call are written in R (reading a date, a flag), the call is the twin. tools/r_twin.ts writes the SDTM records and the expected records of each fixture as CSV and compares the values: 527 values, all the same. The fixtures: fx2, fx4, fx7, fx13 (merge_subject), fx9, fx10 (pick, first and last, with keys), fx11 (summarize: sum, mean, max, min), fx12, fx14 (flag_if_any, else N and blank), fx15, fx23 (flag_first, first and last, with awtdiff and ady as keys), fx17 (pool), fx19, fx23 (windows), fx20, fx25 (visits), fx21, fx22 (baseline, with chg and pchg), fx23 (locf), fx26 (relchange), fx27 (summarize_groups), fx28 (tte with censoring dates and censor_cnsr), fx35 (param_wide).

The RTF#

tlf_print() writes <file>.rtf next to <file>.lst: plain RTF text, the page and the styles of ODS RTF style=journal as the recorded files of SAS have them (landscape, 15840 by 12240 twips, margins 360; titles and footnotes in the header and the footer of each page, bold italic Arial 9.5 pt; the header row of the columns italic and centred with a rule above and below, a header split at | with its pieces one under the other; the cells in Arial 9.5 pt, left, numbers right in 9 characters (BEST9. of the report); a rule under the last row of a page; a new section for each page of together=; the columns that do not fit the page in a second part with the BY columns repeated). The look differs in these ways: SAS sets the width of a column from the width of its text in Arial, here a width is the number of characters times 100 twips plus the padding, so the cell widths differ a little and the break between two parts of a wide listing is set so that it falls where the three recorded listings have it (13000 twips; another listing may break one column earlier or later); the keepn (keep with next) of a row is set on a heading row only; the shading of the journal style (none in the recorded files) is not written. The text of the cells is the same.

The generated R programs#

Sapling writes the programs of this page for a study: sapling r programs makes out/r/programs/<id>.R for each table, with its listings and figures, from the same analysis contract and the same model as the SAS programs. The SAS generator (src/sas.ts) records every step it writes (the header, the population and record steps, each macro call with its parameters in the order of the macro, the titles); src/r_twin.ts writes the same steps as R. The text of the .sas files is never parsed, so the two programs cannot drift: a block of the SAS program is the block of the R program, in the same order, with the same arguments.

Which is whichFolderMade byUsed by
the engine's programs (r-agentic: sma_setup, ars_load, ars_output)out/tlf/programs/sapling tlf programsthe chain, the dashboard, the pilot digest: unchanged
the SAS programs and the macrosout/sas/sapling sas programsDataStep Studio
the readable R programs and the library copyout/r/programs/, out/r/sapling/sapling r programssapling r run, a person who reads or runs them

The readable programs are an additional product, as out/sas/ is. Nothing of out/tlf/, out/ard/ or the chain reads out/r/.

sapling r programs [--no-companions]   out/r/programs/*.R, out/r/sapling/ (copy of r/sapling/), out/r/programs.json
sapling r run [OUTPUT] [--jobs N] [--timeout SECONDS]
                                       Rscript on out/adam/data: out/r/tlf, ard, listings, figures, logs;
                                       N programs at once (default: half the cores, at most 3), each killed after
                                       SECONDS (default 1800); a program that fails or times out is named with its log
sapling r check                        every number of out/ard (the engine) against out/r/ard: out/r/check.md

How a SAS step becomes R: the population and the records are subset(), order() and cbind() where SAS has PROC SORT and a DATA step merge; a condition is an R condition, and a blank is not equal to a value as in SAS (ablfl ne 'Y' is ablfl != "Y" | is.na(ablfl)); %macro(a=1, b=2, out=x) is x <- macro(a = 1, b = 2) (tlf_ for the seven names of the table above), on one line when it is 100 characters or less, else one argument per line with the commas in front and the = aligned; levels, pairs, stats, format are the lists and vectors of "How an argument is given"; TITLE and FOOTNOTE are the title and footnote of tlf_print. out/r/ is the study folder of the R programs (SPL_ROOT, SPL_OUT, SPL_ADAM are set by sapling r run; r.root: of sapling.yml names the folder on another computer).

A macro without an R function is a # NOT MADE line in the program and a NOT MADE line of the command, never a wrong number. The title and footnote of a figure are the title and footnote arguments of the figure functions, drawn as SAS draws TITLE and FOOTNOTE (the titles centred and bold above, the footnotes left below).

sapling run --readable (or sapling tlf run --readable, or tlf.programs: readable in sapling.yml; --jobs and --timeout as above) makes the TLF step of the chain use them: the numbers come from out/r/programs, and the engine lays the tables out from them (ars_output(run = FALSE) on those numbers), so the files the later steps read are the engine's (out/ard/<id>.csv with the place of each result, out/tlf/listings/<id>.lst, ars_results.json). src/r_chain.ts turns a number of a readable program into the engine's result (analysis, operation, the five group keys: all 18392 results of the pilot are rebuilt exactly). The default is not changed.

To make the readable programs the main R programs of the chain later: the chain would call sapling r programs and sapling r run in place of sma_setup/ars_output, and the steps that read out/ard (the dashboard, package, the pilot digest) would read out/r/ard (the same shape, proved equal by sapling r check); the programs would need the R libraries survival and nlme on the machine (as the engine does), the figure titles, and a faster listing writer. Until then sapling r check keeps the two equal.

The conventions of SAS that the new functions follow#

What is not the same as SAS, and what is not proven#

Faults seen in the SAS macros (reported, not changed)#

What is left#

Nothing of the 94 macros is unmade. What is left is proof: the SAS record of the cases marked unproven above (the options of a method that no table or known answer uses), the tte fixtures with events and censors from several data sets, import_xpt on a transport file (with haven installed), the look of the RTF and the figures against the SAS files with a person's eye, and a SAS run of the draw-again cases for the R plot= and par=.