# Verification report — OWR-14298158-012 (Brignall et al.: generating functions of proper subclasses of the separable permutations, Oberwolfach Report 6/2024, Question 5)

Verification date: 2026-10-16.

**Verdict.** The note gives a **complete negative answer** to Question 5 of R. Brignall's abstract (joint work with
M. Bóna, C. Defant, M. Opler and V. Vatter) in Oberwolfach Report 6/2024, p. 292, by one explicit counterexample.
- **Settled.** The class C = Av(2341, 2413, 3142) is a proper subclass of the separable permutations Av(2413, 3142)
  and contains Av(231). Its generating function f = sum_n |C_n| z^n satisfies
  z(1-z+z^2) f^2 - (1-2z+2z^2) f + (1-z)^2 = 0, so f = (1-2z+2z^2 - sqrt(D))/(2z(1-z+z^2)) with
  D = 1-8z+20z^2-24z^3+16z^4-4z^5. D is irreducible over Q and coprime to (1-4z)(1-z)(1-5z), and f is not in
  K = Q(z, sqrt(1-4z), sqrt(1-6z+5z^2)) (Theorem 1.1, Corollary 1.2).
- **Not new mathematics.** The enumeration of C is Theorem 11 of D. Callan, T. Mansour, M. Shattuck, *Wilf
  classification of triples of 4-letter patterns II*, Discrete Math. Theor. Comput. Sci. 19:1 (2017) #6
  (Theorem 41 of arXiv:1605.04969), and the same generating function is in the PermPAL database (entry for the
  symmetric class Av(1432, 2413, 3142)). The note credits both in the abstract, the introduction and the paragraph
  "Scope and priority".
- **Contribution of the note.** The observation that this class answers Question 5 negatively; a self-contained proof
  of the enumeration by the substitution decomposition; an exact certificate (Kummer theory over Q(z)) that f is
  not in K. No printed answer to Question 5 was found (searches below); this is not a proof of novelty, and no
  priority is claimed.
- **Not treated.** Which proper subclasses have generating functions in K; Question 1 and Conjecture 3 of the same
  abstract; the questions of V. Vatter's abstract (Question 4, p. 283, and the different Question 5, p. 284).

Corpus status suggested by this report: solved (answer: no). The note is unrefereed. Three independent
AI-assisted verification runs examined the result; the third one examined the final text of the note and this
package (section "Third independent verification run" below).

## Statement checked
- **Primary source.** R. Brignall, *Enumeration in subclasses of the separable permutations, and related
  questions* (joint work with M. Bóna, C. Defant, M. Opler and V. Vatter), in: Mini-Workshop: Permutation Patterns,
  Oberwolfach Rep. 21 (2024), no. 1, pp. 290–293 (the report occupies pp. 273–308), doi:10.4171/OWR/2024/6.
  Read in the open PDF from EMS Press (MD5 7a3c22729358d9371f645813aeef980b, the same file in all runs).
  - Section 2 of the abstract ("The form of generating functions for subclasses of the separables", pp. 291–292):
    a property noted during the workshop (proper subclasses containing Av(231) or a symmetry have generating
    functions in Q(z, sqrt(1-4z))); the question whether every proper subclass has this property; the report that
    this is false because Av(2143, 2413, 3142) (OEIS A033321, from Brignall–Huczynska–Vatter 2008) has generating
    function 2/(1 + z + sqrt(1-6z+5z^2)); then Question 5, which asks whether "every proper subclass of the
    separable permutations has a generating function that belongs to" Q(z, sqrt(1-4z), sqrt(1-6z+5z^2)).
  - The source gives no answer, conjecture or partial result for Question 5.
  - The same report has a **different** Question 5 in V. Vatter's abstract (p. 284: labelled well-quasi-ordered
    classes and algebraic generating functions). The note always says "Question 5 of Brignall's abstract (p. 292)".
- **Normalisation in the source.** The source writes the generating function of the separable permutations as
  (1 - z - sqrt(1-6z+z^2))/(2z) = 1 + 2z + 6z^2 + 22z^3 + ...; the coefficient of z^n is the number of separable
  permutations of length n+1. Its formula 2/(1 + z + sqrt(1-6z+5z^2)) for Av(2143, 2413, 3142) counts the empty
  permutation. Neither normalisation affects membership in a field containing Q(z).
- **Corpus record.** ulamai/UnsolvedMath, OWR-14298158-012. Its statement field is an extraction of pp. 291–292 in
  which both radicals were lost and the text of Section 3 of the abstract was appended; with the radicals restored
  it is Question 5 of Brignall's abstract. The note quotes the question from the source with both radicals.

## Readings
| Reading | Answer | Where |
|---|---|---|
| Question 5 as posed (all proper subclasses of Av(2413, 3142)) | no | Theorem 1.1, Corollary 1.2 |
| Restricted to proper subclasses containing Av(231) or a symmetry of it | no (C contains Av(231)) | Theorem 1.1(a) |
| Restricted to classes Av(2413, 3142, β) with one extra basis element | no (C is of this form); for β of length at most 3 the generating function is always in K | Corollary 1.2, Section 5.4 |
| Constants C instead of Q, i.e. the field C(z, sqrt(1-4z), sqrt(1-6z+5z^2)) | no | Remark 4.7 |
| "Proper" dropped (the separable permutations themselves) | their generating function is not in K either, so the word "proper" is needed | Section 5.1 |
| Corpus statement read literally (radicals lost, the field becomes Q(z)) | would ask for rationality; not the source's question | Section 1.1 (question quoted from the source) |
| Vatter's Question 5 (p. 284) | a different question; not treated | Section 1.1, Scope and priority |
| Constant term of the generating function, signs of the radicals | irrelevant: f, f - 1, (f - 1)/z generate the same field; the proof only uses that K is generated by square roots of 1-4z and (1-z)(1-5z) | Section 1.1 |

## Results in the paper
- **Theorem 1.1** (a) C is a proper subclass of the separable permutations and contains Av(231); (b) the quadratic
  equation (1) and the closed form (2); (c) D is irreducible over Q, coprime to (1-4z)(1-z)(1-5z), and f is not in K.
  **Corollary 1.2**: the answer to Question 5 is negative, also for the restrictions listed above.
- **Section 2.** Lemma 2.1 (unique sum and skew decompositions, via cuts; no permutation is both sum- and
  skew-decomposable); Lemma 2.2 (an occurrence of a pattern in a skew or direct sum splits into consecutive runs of
  its components); Lemma 2.3 (every separable permutation of length at least 2 is sum- or skew-decomposable; a
  short induction on the length, using the entry n, is included).
- **Section 3.** Lemma 3.1 (C is closed under direct sums); Lemma 3.2 (a skew sum of m >= 2 skew-indecomposable
  permutations lies in C iff all blocks lie in C and all blocks but the last avoid 123); Lemma 3.3 (the separable
  skew-indecomposable 123-avoiders are 1 and δ_a ⊕ δ_b, generating function J = z + z^2/(1-z)^2); Proposition 3.4
  (the specification f = 1/(1-E), I = z + E^2/(1-E), E = z + J I/(1-J)); the elimination giving (3) and (1); the
  identity (Q - 2Pf)^2 = D; Table 1 (counts for n <= 10, and |C_n| up to n = 15); Remark 3.5 (the equation of
  Callan–Mansour–Shattuck from left-to-right maxima is the same quadratic).
- **Section 4.** Lemma 4.1 (an irreducible factor of odd multiplicity prevents a square); Lemma 4.2 (descent in a
  quadratic extension); Lemma 4.3 ([K : Q(z)] = 4); Proposition 4.4 (Kummer: the nonzero elements of Q(z) that are squares
  in K lie in the square classes of 1, 1-4z, (1-z)(1-5z), (1-4z)(1-z)(1-5z)); Corollary 4.5 (membership in K of a
  quadratic power series is decided by the square class of its discriminant); Proposition 4.6 (D irreducible: its
  reduction mod 5 has no root and no irreducible quadratic factor, the ten remainders are listed; D coprime to ab;
  D in none of the four classes); proof of Theorem 1.1(c); Remark 4.7 (only "squarefree and coprime to ab" is used;
  the argument works over C(z)).
- **Section 5.** The Catalan generating function (class of 1-4z) and the source's example (class of 1-6z+5z^2) lie
  in K; the separable generating function does not; the source's own report on the noted property, with which C is
  consistent (not a correction of the source); consistency with Albert–Atkinson–Vatter (rationality) and with
  Albert–Atkinson / Brignall–Huczynska–Vatter (algebraicity); no extra basis element of length at most 3 gives a
  counterexample.

## Computations (sanity checks; programs and outputs in reproducibility/)
No proof depends on a computation.
- **Program of the note** (`reproducibility/check_note.py`, Python 3 with numpy and sympy, about 8 s, 29 checks, all
  PASS; recorded output `check_note_output.txt`; two control checks were made exact after the third verification
  run, see below). (i) Brute force from the definition over all permutations of
  length n <= 10 (every k-subset of positions tested): |C_n| = 1, 2, 6, 21, 77, 290, 1118, 4398, 17595, 71385, the
  sum- and skew-indecomposable members, the separable permutations, Av(231) (Catalan control, and Av(231) ⊆ C),
  Av(2143, 2413, 3142), Lemma 3.3; a generating tree to n = 12. (ii) The specification to order 15 against these
  counts, each equation separately; elimination; (1), (4) and (2) on the series; the factorisation of D, the mod-5
  remainders as printed, gcd(D, ab) = 1, the factorisations of D·c for c in {1, a, b, ab}, with positive and
  negative controls; the normalisation of Section 1.1; Table 1 and the other printed data. (iii) The source's
  example: counts n <= 10 against the series of 2/(1 + z + sqrt(1-6z+5z^2)), its equation and its discriminant.
- **Programs with which the result was first obtained** (not in the package): an enumeration of C by insertion of
  the maximum (n <= 13: 1, 2, 6, 21, 77, 290, 1118, 4398, 17595, 71385, 293042, 1215035, 5081259), a test over all
  permutations (n <= 9), and an exact computation of the specification, of D (factorisations over Q and modulo 3, 5,
  7, 11, 13) and of the square classes. Run B re-ran them from a copy: all nine recorded outputs were reproduced
  (up to timing fields).
- **Programs of the verification runs**: those of runs A and B are not in the package; those of the third run are in
  `reproducibility/independent_run_2/` (see the next section). Counts agree everywhere; the largest are
  |C_14| = 21408350 (runs A, B and the third run), |C_15| = 90786332 (run B and the third run) and
  |C_16| = 387212538 (the third run), equal to the series of the specification.

## Independent verification runs
Two independent verification runs, both AI-assisted, examined the first written version of the result, which
contained the proofs of Lemmas 2.1 and 2.2 and of Sections 3 and 4 of the note (Lemma 2.3 being quoted there from the
literature) and an analytic second argument. Each run wrote its own programs; neither read the other's report.

| Item | Run A | Run B |
|---|---|---|
| Statement and source (Question 5 of Brignall's abstract, p. 292; the other Question 5 on p. 284) | CONFIRMED (source read; fix: always name Brignall's abstract, p. 292, and quote both radicals) | CONFIRMED (source read) |
| Theorem 1.1(a) | CONFIRMED (re-derived) | CONFIRMED; Av(231) ⊆ C checked for n <= 9 |
| Lemmas 2.1, 2.2 (decompositions, blocks) | CONFIRMED (re-derived line by line) | CONFIRMED (proof read) |
| Lemmas 3.1–3.3 | CONFIRMED (re-derived; machine-checked on all separable permutations of length <= 8, 5439 skew-decomposable ones for Lemma 3.2) | CONFIRMED (proof read; exhaustive for length <= 9, 26,232 skew-decomposable ones for Lemma 3.2) |
| Specification (S1)–(S3) | CONFIRMED (re-derived; each equation against brute-force counts of sum- and skew-indecomposables, n <= 14) | CONFIRMED (60 terms; each equation separately against counts, n <= 15) |
| Quadratic (1), D, closed form (2) | CONFIRMED (re-derived; also the left-to-right-maxima equation of CMS rearranged to the same quadratic) | CONFIRMED (sympy; the counts to n = 15 determine the quadratic among equations with coefficients of degree <= 3) |
| Enumeration of C | CONFIRMED (append-last generating tree, n <= 14; all permutations, n <= 11) | CONFIRMED (two C programs: all permutations n <= 11, generating tree n <= 15) |
| D irreducible, gcd(D, ab) = 1 | CONFIRMED (mod 5 exhaustive; gcd(D, D') = gcd(D, ab) = 1; disc(D) = -269056) | CONFIRMED (own factor search mod 5 and mod 11; sympy) |
| f not in K (Kummer argument) | CONFIRMED (re-derived; fix: the wording of the square-class fact) | CONFIRMED (own square-class test with 11 controls) |
| Analytic second argument | valid in substance, needs a continuation step; one false side remark on growth rates (fixes) | correct in substance, needs the path-continuation sentence (recommended) |
| Literature and novelty | enumeration in print (CMS 2017, Theorem 11; arXiv 2016, Theorem 41) and PermPAL; no answer to Question 5 found | the same; one citing work (Semantic Scholar), none in OpenAlex or Crossref; no answer found |
| Sentences about sources | three corrections (normalisation, attribution of algebraicity, a preprint date) and two wording fixes | two corrections (normalisation, description of the source's own report) and additions to the list of checked sources |

Neither run found a wrong statement in the proof of the answer, or a gap that could not be closed.

**Corrections required by the two runs**, all applied in the note:
1. (A1) The question is named "Question 5 of Brignall's abstract (p. 292)" everywhere and quoted with both radicals;
   the different Question 5 of Vatter's abstract (p. 284) is mentioned: Section 1.1, Corollary 1.2, Scope and
   priority.
2. (A2, B1) The normalisation of the separable generating function is stated correctly (coefficient of z^n = number
   of separable permutations of length n+1): Section 1.1; checked by `check_note.py`.
3. (A3) The incorrect side remark on growth rates is deleted (the note makes no growth-rate claim).
4. (A4, B5) The analytic second argument, which needed a continuation step, is omitted; the algebraic proof
   (Section 4) is complete without it.
5. (A5, B3) Algebraicity for classes with finitely many simple permutations is attributed to M. H. Albert and
   M. D. Atkinson, Discrete Math. 300 (2005), doi:10.1016/j.disc.2005.06.016, with Brignall–Huczynska–Vatter (2008)
   as the refinement: Section 5.3.
6. (A6) The square-class lemma is stated for an irreducible factor of odd multiplicity (Lemma 4.1), and the note says
   that only "D squarefree and coprime to ab" is used (Remark 4.7).
7. (A7) Vatter, arXiv:2602.16355: v1 of 18 February 2026, v2 of 25 February 2026 (bibliography).
8. (A8) No statement about what the authors of the question examined; the minimality claim is supported by the
   argument for extra basis elements of length at most 3 (Section 5.4).
9. (A9) No unverifiable search statement is made; the searches listed are those of the logged requests.
10. (B2) The source's own report on the noted property (the source asks whether it always holds and reports that it
    does not) is described accurately; C is presented as consistent with that report, not as a correction:
    Section 5.2. The remark is kept in the paper and is not intended for the corpus notice.
11. (B4) Listed as checked: Hohmeier–Slivken arXiv:2502.19541 (the only citing work found, via Semantic Scholar),
    Brignall–Opler–Vatter arXiv:2602.22089, and Example 6.4 of Brignall–Huczynska–Vatter: Scope and priority.

**Changes made when the note was written, after runs A and B.**
- Lemma 2.3 (separable permutations of length >= 2 are sum- or skew-decomposable) is proved in the note, by
  induction on the length; the first version quoted it.
- Corollary 4.5 (membership in K of a quadratic power series through the square class of its discriminant) and the
  examples of Section 5.1 (Catalan, the source's example, the separable permutations) were added.
- Sections 5.2–5.4 were written in their present form; the consistency remark on Bóna's nonrationality criterion and
  the table of the six one-pattern extensions (which rested on PermPAL equations) were left out.
- The program `check_note.py` was written for the note.
- The final text was then examined by a third independent verification run (next section).

## Third independent verification run (examination of the final text)
A third independent verification run, also AI-assisted, examined the text of the note and this package as if
nothing had been checked before, beginning with the parts added after runs A and B. Its programs were written from
the definitions, independently of `check_note.py`, of the programs with which the result was first obtained and of
those of runs A and B; they are in `reproducibility/independent_run_2/` with their recorded outputs and a README.

| Item | Third run |
|---|---|
| Statement and source | CONFIRMED: Oberwolfach Report 6/2024 read again (pp. 273, 282–284, 286–287, 290–293; rendered pages and text); the 14-word quotation is verbatim, K is displayed with both radicals; Vatter's different Question 5 (p. 284) is set aside; the source's own report on the noted property is described neutrally |
| Theorem 1.1(a), Lemmas 2.1, 2.2 | CONFIRMED (re-derived line by line; Lemma 2.1 on all permutations of length <= 8, Lemma 2.2 on 219,534 pairs) |
| Lemma 2.3 (added after runs A and B) | CONFIRMED (both cases and all sub-cases re-derived, including the ranges of the cuts and the order of values in the forbidden patterns 3142 and 2413; tested on all separable permutations of length <= 9; the closure of {1} under the two sums equals Av(2413, 3142) for n <= 9) |
| Lemmas 3.1–3.3, Proposition 3.4, (3), (1), (4), (2), Remark 3.5 | CONFIRMED (re-derived; Lemma 3.2 on all 13,143 skew-decomposable permutations of length <= 8 and all 26,232 skew-decomposable separable ones of length <= 9; exact series to order 40; Remark 3.5 against the proof of CMS Theorem 11) |
| Enumeration of C, Table 1 | CONFIRMED (generating tree that prepends a new first entry, n <= 16, with 387212538 members of length 16; all permutations, n <= 11; with the sum- and skew-indecomposable members; (S1)–(S3) separately) |
| Lemmas 4.1–4.3, Proposition 4.4, Corollary 4.5 (added after runs A and B), Proposition 4.6, Theorem 1.1(c), Remark 4.7 | CONFIRMED (re-derived; own exact arithmetic over Q, GF(5) and GF(11); the ten quadratics and ten remainders as printed; 16 square-class controls; disc(D) = -269056) |
| Section 5 (rewritten after runs A and B) | CONFIRMED (all equations, discriminants and printed occurrences; the counts of Av(2413, 3142, β) for β of length 3; the case 221 formula of CMS Table 1 against the count of Av(1324, 2413, 3142)) |
| Cited results | CONFIRMED: CMS17 Theorem 11 and Table 1, CMS16 Theorem 41, PermPAL entry, BHV08 abstract and Example 6.4, AAV11 Theorem 4.1 (arXiv version, read in this run); AA05 not readable (publisher page refused the request), used as stated in the BHV08 abstract |
| `check_note.py` | re-run from the archive of version 1: 29 checks PASS, output reproduced apart from the running time; read in full; two control checks tested less than the note says (corrected) |
| Literature and novelty | repeated (below); no printed answer to Question 5 found |

It found no mathematical error. **Corrections required by the third run**, all applied:
1. Abstract (and Zenodo description): "a nonzero element of Q(z) is a square in K only if ...".
2. Verification paragraph: the first two runs found no gap in the proofs given in the note; the one gap they found
   (a missing continuation step) was in the omitted analytic argument (the earlier sentence said they found no gap).
3. Verification paragraph: run B's two programs reached n = 11 (all permutations) and n = 15 (generating tree).
4. The third run is recorded in the Verification paragraph, in this report and in `reproducibility/` (folder
   `independent_run_2/`, README updated); "nine web searches" and the repeated search in Scope and priority.
5. The rationality theorem is cited as Albert–Atkinson–Vatter, Theorem 4.1 (Sections 1.1, 5.2, 5.3, 5.4), with
   arXiv:1007.1014 in the bibliography; "What was read" now says that it was read in the arXiv version and that
   Albert–Atkinson (2005) was not read.
6. `check_note.py`: the controls for 1-4z and 1-6z+5z^2 now test that each lies in the class of a (resp. b) and in no
   other class, as Section 6 of the note says (before, they tested only that each lies in exactly one class); the
   recorded output was regenerated (29 checks, all PASS).
7. Section 5.1: "every power series that is algebraic of degree at most 2 over F".
Also applied (optional): Remark 4.7 gives the reason for the five distinct roots of D (squarefreeness), and one
sentence after the proof of Theorem 1.1(a) derives Corollary 1.2.

## Relation to the literature, novelty and scope
- **Searches (16 October 2026, when the result was first obtained and by runs A and B; all requests anonymous and
  logged).** arXiv API (separable permutations; separable permutations and generating functions; subclasses of the
  separable permutations; the five proposers as authors); zbMATH Open ("separable permutations", 2024–2026: 22
  documents; the proposers as authors); Semantic Scholar (citing works of the report: one, Hohmeier–Slivken
  arXiv:2502.19541, on permuton limits); OpenAlex and Crossref (no citing works); Crossref keyword search; the OEIS
  (the counting sequence 1, 2, 6, 21, 77, 290, 1118, 4398, 17595 is not there; control queries return A006318 and
  A033321); PermPAL (six entries for Av(β, 2413, 3142)); the programme of Permutation Patterns 2025; eight web
  searches. When the note was written, Crossref was queried for the DOIs of the bibliography and the arXiv API for the
  arXiv identifiers.
- **Searches of the third run (16 October 2026; anonymous, logged).** arXiv API: abstracts with "separable
  permutations" and "generating function" (15 results), "subclass(es) of the separable", "separable" and
  "permutation" since 2024 (scanned by title), the proposers as authors since 2024, "pattern" and "algebraic
  generating function" since 2024, 2341, 2413 and 3142 in one abstract (none); Crossref (the report is cited 0 times;
  a keyword query since 2024); OpenAlex (the report, cited 0 times); Semantic Scholar (one citing work,
  arXiv:2502.19541); zbMATH Open (22 documents on separable permutations 2024–2026, 7 with it in the title
  2023–2026); the OEIS (the counting sequence of C is absent; control A033321 found); one web search (nine in all).
  Nothing found states or answers Question 5. The PermPAL entry for C was fetched again; the other five one-pattern
  entries and the home page could not be re-fetched (TLS handshake failure; one attempt per page and a second one
  for the home page; not worked around).
  - Found: the enumeration of C (Callan–Mansour–Shattuck 2017, Theorem 11, and arXiv 2016, Theorem 41; PermPAL,
    specification found on 23 July 2021). Neither the papers nor the database entry discuss separable permutations
    or the field K.
  - Not found: any statement of the negative answer to Question 5, or any examination of K for this class.
  - Examined and not relevant to the question: Brignall–Opler–Vatter arXiv:2602.22089 (linear clique-width, work
    from the same workshop); Vatter arXiv:2602.16355 (a problem collection; its separable items concern separable
    derangements); Brignall–Huczynska–Vatter, Example 6.4 (only Av(2143, 2413, 3142)); Callan–Mansour–Shattuck
    arXiv:1709.04279 (completion of the classification of triples); Bevan arXiv:1510.06328 (the class
    Av(4213, 2143), not a subclass of the separable permutations), which the corpus record's triage cites.
- **What was read.** Brignall's abstract (pp. 290–293) and Vatter's abstract (pp. 282–284) of the report; of
  Callan–Mansour–Shattuck (DMTCS), Section 3.3 with the proof of Theorem 11, and Table 1; of arXiv:1605.04969,
  Section 3.12; the PermPAL entry; the abstract and Example 6.4 of the arXiv version of Brignall–Huczynska–Vatter
  (the journal version was not compared); the texts of arXiv:2602.16355, 2502.19541 and 2602.22089, searched for the
  question; the abstract and Theorem 4.1 of the arXiv version (1007.1014v1) of Albert–Atkinson–Vatter (2011), read
  by the third run (the journal version was not compared). Albert–Atkinson (2005) was not read (the publisher's page
  refused the third run's request); its theorem is used as stated in the abstract of Brignall–Huczynska–Vatter.
  All bibliographic data were checked with Crossref and the arXiv API when the note was written, and again by the
  third run (the five DOIs of the bibliography; the arXiv identifiers 1605.04969, 2602.16355, 2502.19541,
  2602.22089, math/0608391, 2202.07715 and 1007.1014).
- **Caveats.** MathSciNet and Google Scholar were not searched. The PermPAL equations for the other classes
  Av(β, 2413, 3142) with |β| = 4 are not re-proved and are not used. A search that finds nothing is not a proof of
  novelty.
- **Scope.** One explicit counterexample answers the universal question; no characterisation of the subclasses with
  generating functions in K is claimed.

## Public release and license
This paper, its source files, and this verification report are licensed under the Creative Commons Attribution
4.0 International License (CC BY 4.0): https://creativecommons.org/licenses/by/4.0/