After his death, Writer Eduardo Arroyo wrote a biography of Panama Al, titled Panama Al Brown, 1902-1951.

Brown turned pro in 1922 under the guidance of manager Dave Lumiansky. His first fight took place on March 19, 1922, when he beat Jose Moreno by a decision in six at Colon. By his seventh fight, December 13 of that same year, he beat Sailor Patchett by a fifteen round decision, to earn the Isthmus Flyweight title.

On September 22, 1923, he had his first fight abroad, drawing (tying) in four rounds with Johnny Breslin, at New York. He very quickly established a presence upon relocating to New York in 1923.

Brown began campaigning extensively across the United States before he suffered his first loss, at the hands of Jimmy Russo on December 6, 1924, by decision in twelve. He would later avenge that defeat, and he beat Davey Abad and Willie LaMorte before being disqualified in the first round against Frankie Murray on July 11, 1926. Despite that setback, he kept on campaigning successfully and, on November 10 of that year, he knocked out Antoine Merlo in his Paris debut.

An interesting case happened when he fought Gustav Humery, on January 29, 1929. Brown and Humery had previously agreed that they would not salute by touching gloves before the fight, and when the bell rang, Brown struck quickly, breaking Humery's jaw with his first punch and sending him to the floor. With the referee's count of ten seconds, the fight lasted a total of fifteen seconds, one of the quickest knockouts in boxing history.

Brown retained his title nine times and had countless other fights before a rematch with Hummey that ended in disaster: on May 17, 1934, Brown was disqualified in round six at Paris for using illegal tactics. A riot started and Brown suffered several broken bones and was sent into semi-unconsciousness by fans before the police could help him. Twenty minutes later, the locale where the rematch was held had almost been entirely destroyed.

On June 1, 1935, he lost the title to Baltasar "Sangchili" Belenguer Hervás of Spain, by a fifteen round decision, at Valencia, Spain. They had a rematch on March 4, 1938, with Brown avenging his earlier loss with a fifteen round decision, but by then, Sixto Escobar of Puerto Rico had already taken the world Bantamweight championship. His rematch win over Sangchili is believed to be his last great night and, bowing to Cocteau's wishes, Brown vowed to retire after one more fight. That came in 1939 against Valentine Angelmann in Paris (Brown stopped him in eight rounds).

Brown went on fighting until 1942, challenging unsuccessfully for the Panamanian Featherweight title on September 30, 1942, when he drew with Leocadio Torres, but retiring as a winner, defeating Kid Fortune by a decision in ten rounds on December 4 of the same year.

Brown died penniless of tuberculosis in New York City in 1951. He had fainted on 42nd Street. The police thought he was drunk and took him to the station. Eventually he was transferred to Sea View Hospital. He died there on April 11, unaware that not long before, one of the newspapers in Paris had begun talks about organizing a fund drive to pay for his trip home.